diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/advopt.txt | 93 | ||||
| -rw-r--r-- | doc/apis.md (renamed from doc/apis.rst) | 39 | ||||
| -rw-r--r-- | doc/astspec.txt | 533 | ||||
| -rw-r--r-- | doc/backends.md (renamed from doc/backends.rst) | 313 | ||||
| -rw-r--r-- | doc/basicopt.txt | 16 | ||||
| -rw-r--r-- | doc/codeowners.rst | 65 | ||||
| -rw-r--r-- | doc/contributing.md | 798 | ||||
| -rw-r--r-- | doc/contributing.rst | 557 | ||||
| -rw-r--r-- | doc/destructors.md | 801 | ||||
| -rw-r--r-- | doc/destructors.rst | 632 | ||||
| -rw-r--r-- | doc/docgen.md | 891 | ||||
| -rw-r--r-- | doc/docgen.rst | 395 | ||||
| -rw-r--r-- | doc/docgen_sample.nim | 2 | ||||
| -rw-r--r-- | doc/docs.md (renamed from doc/docs.rst) | 24 | ||||
| -rw-r--r-- | doc/docstyle.md (renamed from doc/docstyle.rst) | 90 | ||||
| -rw-r--r-- | doc/drnim.md (renamed from doc/drnim.rst) | 111 | ||||
| -rw-r--r-- | doc/effects.txt | 5 | ||||
| -rw-r--r-- | doc/estp.md | 206 | ||||
| -rw-r--r-- | doc/estp.rst | 195 | ||||
| -rw-r--r-- | doc/filelist.txt | 10 | ||||
| -rw-r--r-- | doc/filters.md | 218 | ||||
| -rw-r--r-- | doc/filters.rst | 214 | ||||
| -rw-r--r-- | doc/gc.rst | 185 | ||||
| -rw-r--r-- | doc/grammar.txt | 140 | ||||
| -rw-r--r-- | doc/hcr.md (renamed from doc/hcr.rst) | 79 | ||||
| -rw-r--r-- | doc/idetools.md (renamed from doc/idetools.rst) | 364 | ||||
| -rw-r--r-- | doc/intern.md | 679 | ||||
| -rw-r--r-- | doc/intern.rst | 786 | ||||
| -rw-r--r-- | doc/koch.md | 91 | ||||
| -rw-r--r-- | doc/koch.rst | 83 | ||||
| -rw-r--r-- | doc/lib.md | 682 | ||||
| -rw-r--r-- | doc/lib.rst | 568 | ||||
| -rw-r--r-- | doc/manual.md (renamed from doc/manual.rst) | 5994 | ||||
| -rw-r--r-- | doc/manual/var_t_return.md (renamed from doc/manual/var_t_return.rst) | 18 | ||||
| -rw-r--r-- | doc/manual_experimental.md | 2669 | ||||
| -rw-r--r-- | doc/manual_experimental.rst | 1846 | ||||
| -rw-r--r-- | doc/manual_experimental_strictnotnil.md | 255 | ||||
| -rw-r--r-- | doc/markdown_rst.md | 349 | ||||
| -rw-r--r-- | doc/mm.md | 95 | ||||
| -rw-r--r-- | doc/nep1.md (renamed from doc/nep1.rst) | 192 | ||||
| -rw-r--r-- | doc/nimc.md | 844 | ||||
| -rw-r--r-- | doc/nimc.rst | 680 | ||||
| -rw-r--r-- | doc/nimdoc.cls | 196 | ||||
| -rw-r--r-- | doc/nimdoc.css | 406 | ||||
| -rw-r--r-- | doc/nimfix.rst | 56 | ||||
| -rw-r--r-- | doc/nimgrep.md | 128 | ||||
| -rw-r--r-- | doc/nimgrep.rst | 50 | ||||
| -rw-r--r-- | doc/nimgrep_cmdline.txt | 136 | ||||
| -rw-r--r-- | doc/niminst.md (renamed from doc/niminst.rst) | 100 | ||||
| -rw-r--r-- | doc/nims.md (renamed from doc/nims.rst) | 212 | ||||
| -rw-r--r-- | doc/nimsuggest.md (renamed from doc/nimsuggest.rst) | 57 | ||||
| -rw-r--r-- | doc/overview.md (renamed from doc/overview.rst) | 2 | ||||
| -rw-r--r-- | doc/packaging.md (renamed from doc/packaging.rst) | 40 | ||||
| -rw-r--r-- | doc/pegdocs.txt | 100 | ||||
| -rw-r--r-- | doc/readme.txt | 4 | ||||
| -rw-r--r-- | doc/refc.md | 156 | ||||
| -rw-r--r-- | doc/regexprs.txt | 16 | ||||
| -rw-r--r-- | doc/rstcommon.rst | 51 | ||||
| -rw-r--r-- | doc/sets_fragment.txt | 76 | ||||
| -rw-r--r-- | doc/spawn.txt | 12 | ||||
| -rw-r--r-- | doc/subexes.txt | 7 | ||||
| -rw-r--r-- | doc/testament.md | 390 | ||||
| -rw-r--r-- | doc/tools.md | 45 | ||||
| -rw-r--r-- | doc/tools.rst | 39 | ||||
| -rw-r--r-- | doc/tut1.md (renamed from doc/tut1.rst) | 1205 | ||||
| -rw-r--r-- | doc/tut2.md (renamed from doc/tut2.rst) | 302 | ||||
| -rw-r--r-- | doc/tut3.md (renamed from doc/tut3.rst) | 239 |
67 files changed, 16237 insertions, 10595 deletions
diff --git a/doc/advopt.txt b/doc/advopt.txt index f5b8c9ec9..e4d11081a 100644 --- a/doc/advopt.txt +++ b/doc/advopt.txt @@ -4,17 +4,22 @@ Advanced commands: //compileToOC, objc compile project to Objective C code //js compile project to Javascript //e run a Nimscript file + //md2html convert a Markdown file to HTML + use `--docCmd:skip` to skip compiling snippets //rst2html convert a reStructuredText file to HTML use `--docCmd:skip` to skip compiling snippets - //rst2tex convert a reStructuredText file to TeX + //md2tex convert a Markdown file to LaTeX + //rst2tex convert a reStructuredText file to LaTeX + //doc2tex extract the documentation to a LaTeX file //jsondoc extract the documentation to a json file //ctags create a tags file //buildIndex build an index for the whole documentation //genDepend generate a DOT file containing the - module dependency graph + module dependency graph //dump dump all defined conditionals and search paths - see also: --dump.format:json (useful with: ` | jq`) - //check checks the project for syntax and semantic + see also: --dump.format:json (useful with: `| jq`) + //check checks the project for syntax and semantics + (can be combined with --defusages) Runtime checks (see -x): --objChecks:on|off turn obj conversion checks on|off @@ -25,32 +30,53 @@ Runtime checks (see -x): --floatChecks:on|off turn all floating point (NaN/Inf) checks on|off --nanChecks:on|off turn NaN checks on|off --infChecks:on|off turn Inf checks on|off - --refChecks:on|off turn ref checks on|off (only for --newruntime) Advanced options: + --defusages:FILE,LINE,COL + find the definition and all usages of a symbol -o:FILE, --out:FILE set the output filename --outdir:DIR set the path where the output file will be written --usenimcache will use `outdir=$$nimcache`, whichever it resolves to after all options have been processed --stdout:on|off output to stdout --colors:on|off turn compiler messages coloring on|off - --listFullPaths:on|off list full paths in messages + --filenames:abs|canonical|legacyRelProj + customize how filenames are rendered in compiler messages, + defaults to `abs` (absolute) + --processing:dots|filenames|off + show files as they're being processed by nim compiler + --unitsep:on|off use the ASCII unit separator (31) between error + messages, useful for IDE-like tooling + --declaredLocs:on|off show declaration locations in messages + --spellSuggest:num show at most `num >= 0` spelling suggestions on typos. + if `num` is not specified (or `auto`), return + an implementation defined set of suggestions. + --hints:on|off|list. `on|off` enables or disables hints. + `list` reports which hints are selected. + --hint:X:on|off turn specific hint X on|off. `hint:X` means `hint:X:on`, + as with similar flags. `all` is the set of all hints + (only `all:off` is supported). + --hintAsError:X:on|off turn specific hint X into an error on|off -w:on|off|list, --warnings:on|off|list - turn all warnings on|off or list all available - --warning[X]:on|off turn specific warning X on|off - --hints:on|off|list turn all hints on|off or list all available - --hint[X]:on|off turn specific hint X on|off - --warningAsError[X]:on|off + `on|off` enables or disables warnings. + `list` reports which warnings are selected. + --warning:X:on|off turn specific warning X on|off. `warning:X` means `warning:X:on`, + as with similar flags. `all` is the set of all warning + (only `all:off` is supported). + --warningAsError:X:on|off turn specific warning X into an error on|off --styleCheck:off|hint|error produce hints or errors for Nim identifiers that do not adhere to Nim's official style guide https://nim-lang.org/docs/nep1.html + --styleCheck:usages only enforce consistent spellings of identifiers, + do not enforce the style on declarations --showAllMismatches:on|off show all mismatching candidates in overloading resolution --lib:PATH set the system library path --import:PATH add an automatically imported module + see also `patchFile` in nimscript which offers more flexibility. --include:PATH add an automatically included module --nimcache:PATH set the path used for generated files see also https://nim-lang.org/docs/nimc.html#compiler-usage-generated-c-code-directory @@ -59,8 +85,8 @@ Advanced options: --noMain:on|off do not generate a main procedure --genScript:on|off generate a compile script (in the 'nimcache' subdirectory named 'compile_$$project$$scriptext'), + and a '.deps' file containing the dependencies; implies --compileOnly - --genDeps:on|off generate a '.deps' file containing the dependencies --os:SYMBOL set the target operating system (cross-compilation) --cpu:SYMBOL set the target processor (cross-compilation) --debuginfo:on|off enables debug information @@ -71,7 +97,7 @@ Advanced options: --clibdir:DIR modify the linker library search path --clib:LIBNAME link an additional C library (you should omit platform-specific extensions) - --project document the whole project (doc2) + --project document the whole project (doc) --docRoot:path `nim doc --docRoot:/foo --project --outdir:docs /foo/sub/main.nim` generates: docs/sub/main.html if path == @pkg, will use nimble file enclosing dir @@ -79,41 +105,37 @@ Advanced options: if path == @default (the default and most useful), will use best match among @pkg,@path. if these are nonexistent, will use project path - -b, --backend:c|cpp|js|objc sets backend to use with commands like `nim doc` or `nim r` + -b, --backend:c|cpp|js|objc + sets backend to use with commands like `nim doc` or `nim r` --docCmd:cmd if `cmd == skip`, skips runnableExamples - else, runs runnableExamples with given options, eg: + else, runs runnableExamples with given options, e.g.: `--docCmd:"-d:foo --threads:on"` - --docSeeSrcUrl:url activate 'see source' for doc and doc2 commands + --docSeeSrcUrl:url activate 'see source' for doc command (see doc.item.seesrc in config/nimdoc.cfg) --docInternal also generate documentation for non-exported symbols --lineDir:on|off generation of #line directive on|off --embedsrc:on|off embeds the original source code as comments in the generated output - --threadanalysis:on|off turn thread analysis on|off --tlsEmulation:on|off turn thread local storage emulation on|off - --taintMode:on|off turn taint mode on|off --implicitStatic:on|off turn implicit compile time evaluation on|off --trmacros:on|off turn term rewriting macros on|off --multimethods:on|off turn multi-methods on|off - --memTracker:on|off turn memory tracker on|off --hotCodeReloading:on|off turn support for hot code reloading on|off --excessiveStackTrace:on|off stack traces use full file paths --stackTraceMsgs:on|off enable user defined stack frame msgs via `setFrameMsg` - --nilseqs:on|off allow 'nil' for strings/seqs for - backwards compatibility - --seqsv2:on|off use the new string/seq implementation based on - destructors --skipCfg:on|off do not read the nim installation's configuration file --skipUserCfg:on|off do not read the user's configuration file --skipParentCfg:on|off do not read the parent dirs' configuration files --skipProjCfg:on|off do not read the project's configuration file - --gc:refc|arc|orc|markAndSweep|boehm|go|none|regions - select the GC to use; default is 'refc' + --mm:orc|arc|refc|markAndSweep|boehm|go|none|regions + select which memory management to use; default is 'orc' --exceptions:setjmp|cpp|goto|quirky select the exception handling implementation - --index:on|off turn index file generation on|off + --index:on|off|only docgen: turn index file generation on|off (`only` means + not generate output files like HTML) + --noImportdoc:on|off turn loading documentation ``.idx`` files on|off --putenv:key=value set an environment variable --NimblePath:PATH add a path for Nimble support --noNimblePath deactivate the Nimble path @@ -121,7 +143,11 @@ Advanced options: --cppCompileToNamespace:namespace use the provided namespace for the generated C++ code, if no namespace is provided "Nim" will be used + --nimMainPrefix:prefix use `{prefix}NimMain` instead of `NimMain` in the produced + C/C++ code --expandMacro:MACRO dump every generated AST from MACRO + --expandArc:PROCNAME show how PROCNAME looks like after diverse optimizations + before the final backend phase (mostly ARC/ORC specific) --excludePath:PATH exclude a path from the list of search paths --dynlibOverride:SYMBOL marks SYMBOL so that dynlib:SYMBOL has no effect and can be statically linked instead; @@ -143,12 +169,11 @@ Advanced options: enable experimental language feature --legacy:$2 enable obsolete/legacy language feature - --useVersion:1.0 emulate Nim version X of the Nim compiler - --newruntime use an alternative runtime that uses destructors - and that uses a shared heap via -d:useMalloc - --profiler:on|off enable profiling; requires `import nimprof`, and - works better with `--stackTrace:on` - see also https://nim-lang.github.io/Nim/estp.html - --benchmarkVM:on|off enable benchmarking of VM code with cpuTime() - --sinkInference:on|off en-/disable sink parameter inference (default: on) + --benchmarkVM:on|off turn benchmarking of VM code with cpuTime() on|off + --profileVM:on|off turn compile time VM profiler on|off --panics:on|off turn panics into process terminations (default: off) + --deepcopy:on|off enable 'system.deepCopy' for ``--mm:arc|orc`` + --jsbigint64:on|off toggle the use of BigInt for 64-bit integers for + the JavaScript backend (default: on) + --nimBasePattern:nimbase.h + allows to specify a custom pattern for `nimbase.h` diff --git a/doc/apis.rst b/doc/apis.md index d0327c76e..f0b8c93e5 100644 --- a/doc/apis.rst +++ b/doc/apis.md @@ -2,8 +2,11 @@ API naming design ================= +.. default-role:: code +.. include:: rstcommon.rst + The API is designed to be **easy to use** and consistent. Ease of use is -measured by the number of calls to achieve a concrete high level action. +measured by the number of calls to achieve a concrete high-level action. Naming scheme @@ -15,25 +18,25 @@ been renamed to fit this scheme. The ultimate goal is that the programmer can *guess* a name. -------------------- ------------ -------------------------------------- +=================== ============ ====================================== English word To use Notes -------------------- ------------ -------------------------------------- -initialize initT ``init`` is used to create a - value type ``T`` -new newP ``new`` is used to create a - reference type ``P`` +=================== ============ ====================================== +initialize initT `init` is used to create a + value type `T` +new newP `new` is used to create a + reference type `P` find find should return the position where something was found; for a bool result - use ``contains`` -contains contains often short for ``find() >= 0`` -append add use ``add`` instead of ``append`` + use `contains` +contains contains often short for `find() >= 0` +append add use `add` instead of `append` compare cmp should return an int with the - ``< 0`` ``== 0`` or ``> 0`` semantics; - for a bool result use ``sameXYZ`` -put put, ``[]=`` consider overloading ``[]=`` for put -get get, ``[]`` consider overloading ``[]`` for get; - consider to not use ``get`` as a - prefix: ``len`` instead of ``getLen`` + `< 0` `== 0` or `> 0` semantics; + for a bool result use `sameXYZ` +put put, `[]=` consider overloading `[]=` for put +get get, `[]` consider overloading `[]` for get; + consider to not use `get` as a + prefix: `len` instead of `getLen` length len also used for *number of elements* size size, len size should refer to a byte size capacity cap @@ -44,7 +47,7 @@ delete delete, del del is supposed to be faster than delete, because it does not keep the order; delete keeps the order remove delete, del inconsistent right now -remove-and-return pop ``Table``/``TableRef`` alias to ``take`` +remove-and-return pop `Table`/`TableRef` alias to `take` include incl exclude excl command cmd @@ -79,4 +82,4 @@ literal lit string str identifier ident indentation indent -------------------- ------------ -------------------------------------- +=================== ============ ====================================== diff --git a/doc/astspec.txt b/doc/astspec.txt index ab2fbcacc..7a7053a2d 100644 --- a/doc/astspec.txt +++ b/doc/astspec.txt @@ -6,8 +6,7 @@ The AST consists of nodes (``NimNode``) with a variable number of children. Each node has a field named ``kind`` which describes what the node contains: -.. code-block:: nim - + ```nim type NimNodeKind = enum ## kind of a node; only explanatory nnkNone, ## invalid node kind @@ -32,6 +31,7 @@ contains: strVal: string ## the string literal else: sons: seq[NimNode] ## the node's sons (or children) + ``` For the ``NimNode`` type, the ``[]`` operator has been overloaded: ``n[i]`` is ``n``'s ``i``-th child. @@ -50,9 +50,9 @@ A leaf of the AST often corresponds to a terminal symbol in the concrete syntax. Note that the default ``float`` in Nim maps to ``float64`` such that the default AST for a float is ``nnkFloat64Lit`` as below. ------------------ --------------------------------------------- +================= ============================================= Nim expression Corresponding AST ------------------ --------------------------------------------- +================= ============================================= ``42`` ``nnkIntLit(intVal = 42)`` ``42'i8`` ``nnkInt8Lit(intVal = 42)`` ``42'i16`` ``nnkInt16Lit(intVal = 42)`` @@ -72,7 +72,7 @@ Nim expression Corresponding AST ``nil`` ``nnkNilLit()`` ``myIdentifier`` ``nnkIdent(strVal = "myIdentifier")`` ``myIdentifier`` after lookup pass: ``nnkSym(strVal = "myIdentifier", ...)`` ------------------ --------------------------------------------- +================= ============================================= Identifiers are ``nnkIdent`` nodes. After the name lookup pass these nodes get transferred into ``nnkSym`` nodes. @@ -86,17 +86,19 @@ Command call Concrete syntax: -.. code-block:: nim + ```nim echo "abc", "xyz" + ``` AST: -.. code-block:: nim + ```nim nnkCommand( nnkIdent("echo"), nnkStrLit("abc"), nnkStrLit("xyz") ) + ``` Call with ``()`` @@ -104,17 +106,19 @@ Call with ``()`` Concrete syntax: -.. code-block:: nim + ```nim echo("abc", "xyz") + ``` AST: -.. code-block:: nim + ```nim nnkCall( nnkIdent("echo"), nnkStrLit("abc"), nnkStrLit("xyz") ) + ``` Infix operator call @@ -122,29 +126,32 @@ Infix operator call Concrete syntax: -.. code-block:: nim + ```nim "abc" & "xyz" + ``` AST: -.. code-block:: nim + ```nim nnkInfix( nnkIdent("&"), nnkStrLit("abc"), nnkStrLit("xyz") ) + ``` Note that with multiple infix operators, the command is parsed by operator precedence. Concrete syntax: -.. code-block:: nim + ```nim 5 + 3 * 4 + ``` AST: -.. code-block:: nim + ```nim nnkInfix( nnkIdent("+"), nnkIntLit(5), @@ -154,6 +161,7 @@ AST: nnkIntLit(4) ) ) + ``` As a side note, if you choose to use infix operators in a prefix form, the AST behaves as a @@ -162,12 +170,13 @@ behaves as a Concrete syntax: -.. code-block:: nim + ```nim `+`(3, 4) + ``` AST: -.. code-block:: nim + ```nim nnkCall( nnkAccQuoted( nnkIdent("+") @@ -175,22 +184,25 @@ AST: nnkIntLit(3), nnkIntLit(4) ) + ``` Prefix operator call -------------------- Concrete syntax: -.. code-block:: nim + ```nim ? "xyz" + ``` AST: -.. code-block:: nim + ```nim nnkPrefix( nnkIdent("?"), nnkStrLit("abc") ) + ``` Postfix operator call @@ -201,16 +213,18 @@ Postfix operator call Concrete syntax: -.. code-block:: nim + ```nim identifier* + ``` AST: -.. code-block:: nim + ```nim nnkPostfix( nnkIdent("*"), nnkIdent("identifier") ) + ``` Call with named arguments @@ -218,12 +232,13 @@ Call with named arguments Concrete syntax: -.. code-block:: nim + ```nim writeLine(file=stdout, "hallo") + ``` AST: -.. code-block:: nim + ```nim nnkCall( nnkIdent("writeLine"), nnkExprEqExpr( @@ -232,6 +247,7 @@ AST: ), nnkStrLit("hallo") ) + ``` Call with raw string literal ---------------------------- @@ -242,29 +258,33 @@ This is used, for example, in the ``bindSym`` examples Concrete syntax: -.. code-block:: nim + ```nim echo"abc" + ``` AST: -.. code-block:: nim + ```nim nnkCallStrLit( nnkIdent("echo"), nnkRStrLit("hello") ) + ``` Dereference operator ``[]`` --------------------------- Concrete syntax: -.. code-block:: nim + ```nim x[] + ``` AST: -.. code-block:: nim + ```nim nnkDerefExpr(nnkIdent("x")) + ``` Addr operator @@ -272,13 +292,15 @@ Addr operator Concrete syntax: -.. code-block:: nim + ```nim addr(x) + ``` AST: -.. code-block:: nim + ```nim nnkAddr(nnkIdent("x")) + ``` Cast operator @@ -286,13 +308,15 @@ Cast operator Concrete syntax: -.. code-block:: nim + ```nim cast[T](x) + ``` AST: -.. code-block:: nim + ```nim nnkCast(nnkIdent("T"), nnkIdent("x")) + ``` Object access operator ``.`` @@ -300,13 +324,15 @@ Object access operator ``.`` Concrete syntax: -.. code-block:: nim + ```nim x.y + ``` AST: -.. code-block:: nim + ```nim nnkDotExpr(nnkIdent("x"), nnkIdent("y")) + ``` If you use Nim's flexible calling syntax (as in ``x.len()``), the result is the same as above but wrapped in an ``nnkCall``. @@ -317,31 +343,77 @@ Array access operator ``[]`` Concrete syntax: -.. code-block:: nim + ```nim x[y] + ``` AST: -.. code-block:: nim + ```nim nnkBracketExpr(nnkIdent("x"), nnkIdent("y")) + ``` Parentheses ----------- -Parentheses for affecting operator precedence or tuple construction -are built with the ``nnkPar`` node. +Parentheses for affecting operator precedence use the ``nnkPar`` node. + +Concrete syntax: + + ```nim + (a + b) * c + ``` + +AST: + + ```nim + nnkInfix(nnkIdent("*"), + nnkPar( + nnkInfix(nnkIdent("+"), nnkIdent("a"), nnkIdent("b"))), + nnkIdent("c")) + ``` + +Tuple Constructors +------------------ + +Nodes for tuple construction are built with the ``nnkTupleConstr`` node. Concrete syntax: -.. code-block:: nim - (1, 2, (3)) + ```nim + (1, 2, 3) + (a: 1, b: 2, c: 3) + () + ``` AST: -.. code-block:: nim - nnkPar(nnkIntLit(1), nnkIntLit(2), nnkPar(nnkIntLit(3))) + ```nim + nnkTupleConstr(nnkIntLit(1), nnkIntLit(2), nnkIntLit(3)) + nnkTupleConstr( + nnkExprColonExpr(nnkIdent("a"), nnkIntLit(1)), + nnkExprColonExpr(nnkIdent("b"), nnkIntLit(2)), + nnkExprColonExpr(nnkIdent("c"), nnkIntLit(3))) + nnkTupleConstr() + ``` +Since the one tuple would be syntactically identical to parentheses +with an expression in them, the parser expects a trailing comma for +them. For tuple constructors with field names, this is not necessary. + + ```nim + (1,) + (a: 1) + ``` + +AST: + + ```nim + nnkTupleConstr(nnkIntLit(1)) + nnkTupleConstr( + nnkExprColonExpr(nnkIdent("a"), nnkIntLit(1))) + ``` Curly braces ------------ @@ -350,28 +422,32 @@ Curly braces are used as the set constructor. Concrete syntax: -.. code-block:: nim + ```nim {1, 2, 3} + ``` AST: -.. code-block:: nim + ```nim nnkCurly(nnkIntLit(1), nnkIntLit(2), nnkIntLit(3)) + ``` When used as a table constructor, the syntax is different. Concrete syntax: -.. code-block:: nim + ```nim {a: 3, b: 5} + ``` AST: -.. code-block:: nim + ```nim nnkTableConstr( nnkExprColonExpr(nnkIdent("a"), nnkIntLit(3)), nnkExprColonExpr(nnkIdent("b"), nnkIntLit(5)) ) + ``` Brackets @@ -381,13 +457,15 @@ Brackets are used as the array constructor. Concrete syntax: -.. code-block:: nim + ```nim [1, 2, 3] + ``` AST: -.. code-block:: nim + ```nim nnkBracket(nnkIntLit(1), nnkIntLit(2), nnkIntLit(3)) + ``` Ranges @@ -399,22 +477,24 @@ AST, construction with ``..`` as an infix operator should be used instead. Concrete syntax: -.. code-block:: nim + ```nim 1..3 + ``` AST: -.. code-block:: nim + ```nim nnkInfix( nnkIdent(".."), nnkIntLit(1), nnkIntLit(3) ) + ``` Example code: -.. code-block:: nim - macro genRepeatEcho(): stmt = + ```nim + macro genRepeatEcho() = result = newNimNode(nnkStmtList) var forStmt = newNimNode(nnkForStmt) # generate a for statement @@ -432,6 +512,7 @@ Example code: # 3 # 3 # 3 + ``` If expression @@ -441,17 +522,19 @@ The representation of the ``if`` expression is subtle, but easy to traverse. Concrete syntax: -.. code-block:: nim + ```nim if cond1: expr1 elif cond2: expr2 else: expr3 + ``` AST: -.. code-block:: nim + ```nim nnkIfExpr( nnkElifExpr(cond1, expr1), nnkElifExpr(cond2, expr2), nnkElseExpr(expr3) ) + ``` Documentation Comments ---------------------- @@ -462,19 +545,21 @@ comments are ignored. Concrete syntax: -.. code-block:: nim + ```nim ## This is a comment ## This is part of the first comment stmt1 ## Yet another + ``` AST: -.. code-block:: nim + ```nim nnkCommentStmt() # only appears once for the first two lines! stmt1 nnkCommentStmt() # another nnkCommentStmt because there is another comment # (separate from the first) + ``` Pragmas ------- @@ -485,30 +570,33 @@ objects, but the standalone ``emit`` pragma shows the basics with the AST. Concrete syntax: -.. code-block:: nim + ```nim {.emit: "#include <stdio.h>".} + ``` AST: -.. code-block:: nim + ```nim nnkPragma( nnkExprColonExpr( nnkIdent("emit"), nnkStrLit("#include <stdio.h>") # the "argument" ) ) + ``` As many ``nnkIdent`` appear as there are pragmas between ``{..}``. Note that the declaration of new pragmas is essentially the same: Concrete syntax: -.. code-block:: nim + ```nim {.pragma: cdeclRename, cdecl.} + ``` AST: -.. code-block:: nim + ```nim nnkPragma( nnkExprColonExpr( nnkIdent("pragma"), # this is always first when declaring a new pragma @@ -516,6 +604,7 @@ AST: ), nnkIdent("cdecl") ) + ``` Statements ========== @@ -528,7 +617,7 @@ there is no ``else`` branch, no ``nnkElse`` child exists. Concrete syntax: -.. code-block:: nim + ```nim if cond1: stmt1 elif cond2: @@ -537,16 +626,18 @@ Concrete syntax: stmt3 else: stmt4 + ``` AST: -.. code-block:: nim + ```nim nnkIfStmt( nnkElifBranch(cond1, stmt1), nnkElifBranch(cond2, stmt2), nnkElifBranch(cond3, stmt3), nnkElse(stmt4) ) + ``` When statement @@ -560,13 +651,15 @@ Assignment Concrete syntax: -.. code-block:: nim + ```nim x = 42 + ``` AST: -.. code-block:: nim + ```nim nnkAsgn(nnkIdent("x"), nnkIntLit(42)) + ``` This is not the syntax for assignment when combined with ``var``, ``let``, or ``const``. @@ -576,15 +669,17 @@ Statement list Concrete syntax: -.. code-block:: nim + ```nim stmt1 stmt2 stmt3 + ``` AST: -.. code-block:: nim + ```nim nnkStmtList(stmt1, stmt2, stmt3) + ``` Case statement @@ -592,7 +687,7 @@ Case statement Concrete syntax: -.. code-block:: nim + ```nim case expr1 of expr2, expr3..expr4: stmt1 @@ -602,10 +697,11 @@ Concrete syntax: stmt3 else: stmt4 + ``` AST: -.. code-block:: nim + ```nim nnkCaseStmt( expr1, nnkOfBranch(expr2, nnkRange(expr3, expr4), stmt1), @@ -613,6 +709,7 @@ AST: nnkElifBranch(cond1, stmt3), nnkElse(stmt4) ) + ``` The ``nnkElifBranch`` and ``nnkElse`` parts may be missing. @@ -622,14 +719,16 @@ While statement Concrete syntax: -.. code-block:: nim + ```nim while expr1: stmt1 + ``` AST: -.. code-block:: nim + ```nim nnkWhileStmt(expr1, stmt1) + ``` For statement @@ -637,14 +736,16 @@ For statement Concrete syntax: -.. code-block:: nim + ```nim for ident1, ident2 in expr1: stmt1 + ``` AST: -.. code-block:: nim + ```nim nnkForStmt(ident1, ident2, expr1, stmt1) + ``` Try statement @@ -652,7 +753,7 @@ Try statement Concrete syntax: -.. code-block:: nim + ```nim try: stmt1 except e1, e2: @@ -663,10 +764,11 @@ Concrete syntax: stmt4 finally: stmt5 + ``` AST: -.. code-block:: nim + ```nim nnkTryStmt( stmt1, nnkExceptBranch(e1, e2, stmt2), @@ -674,6 +776,7 @@ AST: nnkExceptBranch(stmt4), nnkFinally(stmt5) ) + ``` Return statement @@ -681,13 +784,15 @@ Return statement Concrete syntax: -.. code-block:: nim + ```nim return expr1 + ``` AST: -.. code-block:: nim + ```nim nnkReturnStmt(expr1) + ``` Yield statement @@ -695,8 +800,9 @@ Yield statement Like ``return``, but with ``nnkYieldStmt`` kind. -.. code-block:: nim + ```nim nnkYieldStmt(expr1) + ``` Discard statement @@ -704,8 +810,9 @@ Discard statement Like ``return``, but with ``nnkDiscardStmt`` kind. -.. code-block:: nim + ```nim nnkDiscardStmt(expr1) + ``` Continue statement @@ -713,26 +820,30 @@ Continue statement Concrete syntax: -.. code-block:: nim + ```nim continue + ``` AST: -.. code-block:: nim + ```nim nnkContinueStmt() + ``` Break statement --------------- Concrete syntax: -.. code-block:: nim + ```nim break otherLocation + ``` AST: -.. code-block:: nim + ```nim nnkBreakStmt(nnkIdent("otherLocation")) + ``` If ``break`` is used without a jump-to location, ``nnkEmpty`` replaces ``nnkIdent``. @@ -741,13 +852,15 @@ Block statement Concrete syntax: -.. code-block:: nim + ```nim block name: + ``` AST: -.. code-block:: nim + ```nim nnkBlockStmt(nnkIdent("name"), nnkStmtList(...)) + ``` A ``block`` doesn't need an name, in which case ``nnkEmpty`` is used. @@ -756,18 +869,20 @@ Asm statement Concrete syntax: -.. code-block:: nim + ```nim asm """ some asm """ + ``` AST: -.. code-block:: nim + ```nim nnkAsmStmt( nnkEmpty(), # for pragmas nnkTripleStrLit("some asm"), ) + ``` Import section -------------- @@ -777,37 +892,42 @@ on what keywords are present. Let's start with the simplest form. Concrete syntax: -.. code-block:: nim - import math + ```nim + import std/math + ``` AST: -.. code-block:: nim + ```nim nnkImportStmt(nnkIdent("math")) + ``` With ``except``, we get ``nnkImportExceptStmt``. Concrete syntax: -.. code-block:: nim - import math except pow + ```nim + import std/math except pow + ``` AST: -.. code-block:: nim + ```nim nnkImportExceptStmt(nnkIdent("math"),nnkIdent("pow")) + ``` -Note that ``import math as m`` does not use a different node; rather, +Note that ``import std/math as m`` does not use a different node; rather, we use ``nnkImportStmt`` with ``as`` as an infix operator. Concrete syntax: -.. code-block:: nim - import strutils as su + ```nim + import std/strutils as su + ``` AST: -.. code-block:: nim + ```nim nnkImportStmt( nnkInfix( nnkIdent("as"), @@ -815,6 +935,7 @@ AST: nnkIdent("su") ) ) + ``` From statement -------------- @@ -823,15 +944,17 @@ If we use ``from ... import``, the result is different, too. Concrete syntax: -.. code-block:: nim - from math import pow + ```nim + from std/math import pow + ``` AST: -.. code-block:: nim + ```nim nnkFromStmt(nnkIdent("math"), nnkIdent("pow")) + ``` -Using ``from math as m import pow`` works identically to the ``as`` modifier +Using ``from std/math as m import pow`` works identically to the ``as`` modifier with the ``import`` statement, but wrapped in ``nnkFromStmt``. Export statement @@ -842,26 +965,30 @@ the ``export`` syntax is pretty straightforward. Concrete syntax: -.. code-block:: nim + ```nim export unsigned + ``` AST: -.. code-block:: nim + ```nim nnkExportStmt(nnkIdent("unsigned")) + ``` Similar to the ``import`` statement, the AST is different for ``export ... except``. Concrete syntax: -.. code-block:: nim + ```nim export math except pow # we're going to implement our own exponentiation + ``` AST: -.. code-block:: nim + ```nim nnkExportExceptStmt(nnkIdent("math"),nnkIdent("pow")) + ``` Include statement ----------------- @@ -870,25 +997,28 @@ Like a plain ``import`` statement but with ``nnkIncludeStmt``. Concrete syntax: -.. code-block:: nim + ```nim include blocks + ``` AST: -.. code-block:: nim + ```nim nnkIncludeStmt(nnkIdent("blocks")) + ``` Var section ----------- Concrete syntax: -.. code-block:: nim + ```nim var a = 3 + ``` AST: -.. code-block:: nim + ```nim nnkVarSection( nnkIdentDefs( nnkIdent("a"), @@ -896,6 +1026,7 @@ AST: nnkIntLit(3), ) ) + ``` Note that either the second or third (or both) parameters above must exist, as the compiler needs to know the type somehow (which it can infer from @@ -913,12 +1044,13 @@ This is equivalent to ``var``, but with ``nnkLetSection`` rather than Concrete syntax: -.. code-block:: nim + ```nim let a = 3 + ``` AST: -.. code-block:: nim + ```nim nnkLetSection( nnkIdentDefs( nnkIdent("a"), @@ -926,18 +1058,20 @@ AST: nnkIntLit(3), ) ) + ``` Const section ------------- Concrete syntax: -.. code-block:: nim + ```nim const a = 3 + ``` AST: -.. code-block:: nim + ```nim nnkConstSection( nnkConstDef( # not nnkConstDefs! nnkIdent("a"), @@ -945,6 +1079,7 @@ AST: nnkIntLit(3), # required in a const declaration! ) ) + ``` Type section ------------ @@ -954,12 +1089,13 @@ and ``const``. Concrete syntax: -.. code-block:: nim + ```nim type A = int + ``` AST: -.. code-block:: nim + ```nim nnkTypeSection( nnkTypeDef( nnkIdent("A"), @@ -967,18 +1103,20 @@ AST: nnkIdent("int") ) ) + ``` Declaring ``distinct`` types is similar, with the last ``nnkIdent`` wrapped in ``nnkDistinctTy``. Concrete syntax: -.. code-block:: nim + ```nim type MyInt = distinct int + ``` AST: -.. code-block:: nim + ```nim # ... nnkTypeDef( nnkIdent("MyInt"), @@ -987,17 +1125,19 @@ AST: nnkIdent("int") ) ) + ``` If a type section uses generic parameters, they are treated here: Concrete syntax: -.. code-block:: nim + ```nim type A[T] = expr1 + ``` AST: -.. code-block:: nim + ```nim nnkTypeSection( nnkTypeDef( nnkIdent("A"), @@ -1012,6 +1152,7 @@ AST: expr1, ) ) + ``` Note that not all ``nnkTypeDef`` utilize ``nnkIdent`` as their parameter. One of the most common uses of type declarations @@ -1019,12 +1160,13 @@ is to work with objects. Concrete syntax: -.. code-block:: nim + ```nim type IO = object of RootObj + ``` AST: -.. code-block:: nim + ```nim # ... nnkTypeDef( nnkIdent("IO"), @@ -1033,33 +1175,43 @@ AST: nnkEmpty(), # no pragmas here nnkOfInherit( nnkIdent("RootObj") # inherits from RootObj - ) + ), nnkEmpty() ) ) + ``` Nim's object syntax is rich. Let's take a look at an involved example in its entirety to see some of the complexities. Concrete syntax: -.. code-block:: nim - type Obj[T] = object {.inheritable.} + ```nim + type Obj[T] {.inheritable.} = object name: string case isFat: bool of true: m: array[100_000, T] of false: m: array[10, T] + ``` AST: -.. code-block:: nim + ```nim # ... + nnkPragmaExpr( + nnkIdent("Obj"), + nnkPragma(nnkIdent("inheritable")) + ), + nnkGenericParams( + nnkIdentDefs( + nnkIdent("T"), + nnkEmpty(), + nnkEmpty()) + ), nnkObjectTy( - nnkPragma( - nnkIdent("inheritable") - ), + nnkEmpty(), nnkEmpty(), nnkRecList( # list of object parameters nnkIdentDefs( @@ -1103,55 +1255,61 @@ AST: ) ) ) + ``` Using an ``enum`` is similar to using an ``object``. Concrete syntax: -.. code-block:: nim + ```nim type X = enum First + ``` AST: -.. code-block:: nim + ```nim # ... nnkEnumTy( nnkEmpty(), nnkIdent("First") # you need at least one nnkIdent or the compiler complains ) + ``` The usage of ``concept`` (experimental) is similar to objects. Concrete syntax: -.. code-block:: nim + ```nim type Con = concept x,y,z (x & y & z) is string + ``` AST: -.. code-block:: nim + ```nim # ... nnkTypeClassTy( # note this isn't nnkConceptTy! - nnkArglist( + nnkArgList( # ... idents for x, y, z ) # ... ) + ``` Static types, like ``static[int]``, use ``nnkIdent`` wrapped in ``nnkStaticTy``. Concrete syntax: -.. code-block:: nim + ```nim type A[T: static[int]] = object + ``` AST: -.. code-block:: nim + ```nim # ... within nnkGenericParams nnkIdentDefs( nnkIdent("T"), @@ -1161,13 +1319,14 @@ AST: nnkEmpty() ) # ... + ``` In general, declaring types mirrors this syntax (i.e., ``nnkStaticTy`` for ``static``, etc.). Examples follow (exceptions marked by ``*``): -------------- --------------------------------------------- +============= ============================================= Nim type Corresponding AST -------------- --------------------------------------------- +============= ============================================= ``static`` ``nnkStaticTy`` ``tuple`` ``nnkTupleTy`` ``var`` ``nnkVarTy`` @@ -1180,7 +1339,7 @@ Nim type Corresponding AST ``proc`` ``nnkProcTy`` ``iterator`` ``nnkIteratorTy`` ``object`` ``nnkObjectTy`` -------------- --------------------------------------------- +============= ============================================= Take special care when declaring types as ``proc``. The behavior is similar to ``Procedure declaration``, below, but does not treat ``nnkGenericParams``. @@ -1188,12 +1347,13 @@ Generic parameters are treated in the type, not the ``proc`` itself. Concrete syntax: -.. code-block:: nim - type MyProc[T] = proc(x: T) + ```nim + type MyProc[T] = proc(x: T) {.nimcall.} + ``` AST: -.. code-block:: nim + ```nim # ... nnkTypeDef( nnkIdent("MyProc"), @@ -1203,38 +1363,75 @@ AST: nnkProcTy( # behaves like a procedure declaration from here on nnkFormalParams( # ... - ) + ), + nnkPragma(nnkIdent("nimcall")) ) ) + ``` The same syntax applies to ``iterator`` (with ``nnkIteratorTy``), but *does not* apply to ``converter`` or ``template``. +Type class versions of these nodes generally share the same node kind but +without any child nodes. The ``tuple`` type class is represented by +``nnkTupleClassTy``, while a ``proc`` or ``iterator`` type class with pragmas +has an ``nnkEmpty`` node in place of the ``nnkFormalParams`` node of a +concrete ``proc`` or ``iterator`` type node. + + ```nim + type TypeClass = proc {.nimcall.} | ref | tuple + ``` + +AST: + + ```nim + nnkTypeDef( + nnkIdent("TypeClass"), + nnkEmpty(), + nnkInfix( + nnkIdent("|"), + nnkProcTy( + nnkEmpty(), + nnkPragma(nnkIdent("nimcall")) + ), + nnkInfix( + nnkIdent("|"), + nnkRefTy(), + nnkTupleClassTy() + ) + ) + ) + ``` + Mixin statement --------------- Concrete syntax: -.. code-block:: nim + ```nim mixin x + ``` AST: -.. code-block:: nim + ```nim nnkMixinStmt(nnkIdent("x")) + ``` Bind statement -------------- Concrete syntax: -.. code-block:: nim + ```nim bind x + ``` AST: -.. code-block:: nim + ```nim nnkBindStmt(nnkIdent("x")) + ``` Procedure declaration --------------------- @@ -1244,18 +1441,21 @@ a feel for how procedure calls are broken down. Concrete syntax: -.. code-block:: nim + ```nim proc hello*[T: SomeInteger](x: int = 3, y: float32): int {.inline.} = discard + ``` AST: -.. code-block:: nim + ```nim nnkProcDef( nnkPostfix(nnkIdent("*"), nnkIdent("hello")), # the exported proc name nnkEmpty(), # patterns for term rewriting in templates and macros (not procs) nnkGenericParams( # generic type parameters, like with type declaration nnkIdentDefs( - nnkIdent("T"), nnkIdent("SomeInteger") + nnkIdent("T"), + nnkIdent("SomeInteger"), + nnkEmpty() ) ), nnkFormalParams( @@ -1275,6 +1475,7 @@ AST: nnkEmpty(), # reserved slot for future use nnkStmtList(nnkDiscardStmt(nnkEmpty())) # the meat of the proc ) + ``` There is another consideration. Nim has flexible type identification for its procs. Even though ``proc(a: int, b: int)`` and ``proc(a, b: int)`` @@ -1282,12 +1483,13 @@ are equivalent in the code, the AST is a little different for the latter. Concrete syntax: -.. code-block:: nim + ```nim proc(a, b: int) + ``` AST: -.. code-block:: nim + ```nim # ...AST as above... nnkFormalParams( nnkEmpty(), # no return here @@ -1299,24 +1501,27 @@ AST: ) ), # ... + ``` When a procedure uses the special ``var`` type return variable, the result is different from that of a var section. Concrete syntax: -.. code-block:: nim + ```nim proc hello(): var int + ``` AST: -.. code-block:: nim + ```nim # ... nnkFormalParams( nnkVarTy( nnkIdent("int") ) ) + ``` Iterator declaration -------------------- @@ -1326,17 +1531,19 @@ replacing ``nnkProcDef``. Concrete syntax: -.. code-block:: nim + ```nim iterator nonsense[T](x: seq[T]): float {.closure.} = ... + ``` AST: -.. code-block:: nim + ```nim nnkIteratorDef( nnkIdent("nonsense"), nnkEmpty(), ... ) + ``` Converter declaration --------------------- @@ -1345,16 +1552,18 @@ A converter is similar to a proc. Concrete syntax: -.. code-block:: nim + ```nim converter toBool(x: float): bool + ``` AST: -.. code-block:: nim + ```nim nnkConverterDef( nnkIdent("toBool"), # ... ) + ``` Template declaration -------------------- @@ -1367,12 +1576,13 @@ the ``nnkEmpty()`` as the second argument to ``nnkProcDef`` and Concrete syntax: -.. code-block:: nim + ```nim template optOpt{expr1}(a: int): int + ``` AST: -.. code-block:: nim + ```nim nnkTemplateDef( nnkIdent("optOpt"), nnkStmtList( # instead of nnkEmpty() @@ -1380,6 +1590,7 @@ AST: ), # follows like a proc or iterator ) + ``` If the template does not have types for its parameters, the type identifiers inside ``nnkFormalParams`` just becomes ``nnkEmpty``. @@ -1390,6 +1601,18 @@ Macro declaration Macros behave like templates, but ``nnkTemplateDef`` is replaced with ``nnkMacroDef``. +Hidden Standard Conversion +-------------------------- + + ```nim + var f: float = 1 + ``` + +The type of "f" is ``float`` but the type of "1" is actually ``int``. Inserting +``int`` into a ``float`` is a type error. Nim inserts the ``nnkHiddenStdConv`` +node around the ``nnkIntLit`` node so that the new node has the correct type of +``float``. This works for any auto converted nodes and makes the conversion +explicit. Special node kinds ================== diff --git a/doc/backends.rst b/doc/backends.md index e46941a12..9f0c54835 100644 --- a/doc/backends.rst +++ b/doc/backends.md @@ -5,27 +5,33 @@ :Author: Puppet Master :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst +.. no syntax highlighting here by default: + .. contents:: - "Heresy grows from idleness." -- Unknown. + +> "Heresy grows from idleness." -- Unknown. Introduction ============ -The `Nim Compiler User Guide <nimc.html>`_ documents the typical -compiler invocation, using the ``compile`` or ``c`` command to transform a +The [Nim Compiler User Guide](nimc.html) documents the typical +compiler invocation, using the `compile`:option: +or `c`:option: command to transform a ``.nim`` file into one or more ``.c`` files which are then compiled with the -platform's C compiler into a static binary. However there are other commands -to compile to C++, Objective-C or JavaScript. This document tries to +platform's C compiler into a static binary. However, there are other commands +to compile to C++, Objective-C, or JavaScript. This document tries to concentrate in a single place all the backend and interfacing options. The Nim compiler supports mainly two backend families: the C, C++ and -Objective-C targets and the JavaScript target. `The C like targets -<#backends-the-c-like-targets>`_ creates source files which can be compiled -into a library or a final executable. `The JavaScript target -<#backends-the-javascript-target>`_ can generate a ``.js`` file which you -reference from an HTML file or create a `standalone nodejs program -<http://nodejs.org>`_. +Objective-C targets and the JavaScript target. [The C like targets]( +#backends-the-c-like-targets) creates source files that can be compiled +into a library or a final executable. [The JavaScript target]( +#backends-the-javascript-target) can generate a ``.js`` file which you +reference from an HTML file or create a [standalone Node.js program]( +http://nodejs.org). On top of generating libraries or standalone applications, Nim offers bidirectional interfacing with the backend targets through generic and @@ -40,31 +46,33 @@ The C like targets The commands to compile to either C, C++ or Objective-C are: - //compileToC, cc compile project with C code generator - //compileToCpp, cpp compile project to C++ code - //compileToOC, objc compile project to Objective C code +//compileToC, cc compile project with C code generator +//compileToCpp, cpp compile project to C++ code +//compileToOC, objc compile project to Objective C code The most significant difference between these commands is that if you look into the ``nimcache`` directory you will find ``.c``, ``.cpp`` or ``.m`` files, other than that all of them will produce a native binary for your -project. This allows you to take the generated code and place it directly -into a project using any of these languages. Here are some typical command -line invocations:: +project. This allows you to take the generated code and place it directly +into a project using any of these languages. Here are some typical command- +line invocations: - $ nim c hallo.nim - $ nim cpp hallo.nim - $ nim objc hallo.nim + ```cmd + nim c hallo.nim + nim cpp hallo.nim + nim objc hallo.nim + ``` The compiler commands select the target backend, but if needed you can -`specify additional switches for cross compilation -<nimc.html#cross-compilation>`_ to select the target CPU, operative system +[specify additional switches for cross-compilation]( +nimc.html#crossminuscompilation) to select the target CPU, operative system or compiler/linker commands. The JavaScript target --------------------- -Nim can also generate `JavaScript`:idx: code through the ``js`` command. +Nim can also generate `JavaScript`:idx: code through the `js`:option: command. Nim targets JavaScript 1.5 which is supported by any widely used browser. Since JavaScript does not have a portable means to include another module, @@ -73,25 +81,30 @@ Nim just generates a long ``.js`` file. Features or modules that the JavaScript platform does not support are not available. This includes: -* manual memory management (``alloc``, etc.) -* casting and other unsafe operations (``cast`` operator, ``zeroMem``, etc.) +* manual memory management (`alloc`, etc.) +* casting and other unsafe operations (`cast` operator, `zeroMem`, etc.) * file management * OS-specific operations * threading, coroutines * some modules of the standard library -* proper 64 bit integer arithmetic +* proper 64-bit integer arithmetic -To compensate, the standard library has modules `catered to the JS backend -<lib.html#pure-libraries-modules-for-js-backend>`_ +To compensate, the standard library has modules [catered to the JS backend]( +lib.html#pure-libraries-modules-for-js-backend) and more support will come in the future (for instance, Node.js bindings to get OS info). -To compile a Nim module into a ``.js`` file use the ``js`` command; the +To compile a Nim module into a ``.js`` file use the `js`:option: command; the default is a ``.js`` file that is supposed to be referenced in an ``.html`` file. However, you can also run the code with `nodejs`:idx: -(`<http://nodejs.org>`_):: +(http://nodejs.org): + ```cmd nim js -d:nodejs -r examples/hallo.nim + ``` + +If you experience errors saying that `globalThis` is not defined, be +sure to run a recent version of Node.js (at least 12.0). Interfacing @@ -107,85 +120,86 @@ component?). Nim code calling the backend ---------------------------- -Nim code can interface with the backend through the `Foreign function -interface <manual.html#foreign-function-interface>`_ mainly through the -`importc pragma <manual.html#foreign-function-interface-importc-pragma>`_. -The ``importc`` pragma is the *generic* way of making backend symbols available -in Nim and is available in all the target backends (JavaScript too). The C++ -or Objective-C backends have their respective `ImportCpp -<manual.html#implementation-specific-pragmas-importcpp-pragma>`_ and -`ImportObjC <manual.html#implementation-specific-pragmas-importobjc-pragma>`_ +Nim code can interface with the backend through the [Foreign function +interface](manual.html#foreign-function-interface) mainly through the +[importc pragma](manual.html#foreign-function-interface-importc-pragma). +The `importc` pragma is the *generic* way of making backend symbols available +in Nim and is available in all the target backends (JavaScript too). The C++ +or Objective-C backends have their respective [ImportCpp]( +manual.html#implementation-specific-pragmas-importcpp-pragma) and +[ImportObjC](manual.html#implementation-specific-pragmas-importobjc-pragma) pragmas to call methods from classes. Whenever you use any of these pragmas you need to integrate native code into your final binary. In the case of JavaScript this is no problem at all, the -same html file which hosts the generated JavaScript will likely provide other -JavaScript functions which you are importing with ``importc``. +same HTML file which hosts the generated JavaScript will likely provide other +JavaScript functions which you are importing with `importc`. However, for the C like targets you need to link external code either statically or dynamically. The preferred way of integrating native code is to use dynamic linking because it allows you to compile Nim programs without the need for having the related development libraries installed. This is done -through the `dynlib pragma for import -<manual.html#foreign-function-interface-dynlib-pragma-for-import>`_, though -more specific control can be gained using the `dynlib module <dynlib.html>`_. +through the [dynlib pragma for import]( +manual.html#foreign-function-interface-dynlib-pragma-for-import), though +more specific control can be gained using the [dynlib module](dynlib.html). -The `dynlibOverride <nimc.html#dynliboverride>`_ command line switch allows +The [dynlibOverride](nimc.html#dynliboverride) command line switch allows to avoid dynamic linking if you need to statically link something instead. -Nim wrappers designed to statically link source files can use the `compile -pragma <manual.html#implementation-specific-pragmas-compile-pragma>`_ if +Nim wrappers designed to statically link source files can use the [compile +pragma](manual.html#implementation-specific-pragmas-compile-pragma) if there are few sources or providing them along the Nim code is easier than using a system library. Libraries installed on the host system can be linked in with -the `PassL pragma <manual.html#implementation-specific-pragmas-passl-pragma>`_. +the [PassL pragma](manual.html#implementation-specific-pragmas-passl-pragma). -To wrap native code, take a look at the `c2nim tool <https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst>`_ which helps +To wrap native code, take a look at the [c2nim tool]( +https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst) which helps with the process of scanning and transforming header files into a Nim interface. -C invocation example -~~~~~~~~~~~~~~~~~~~~ +### C invocation example Create a ``logic.c`` file with the following content: -.. code-block:: c + ```c int addTwoIntegers(int a, int b) { return a + b; } + ``` Create a ``calculator.nim`` file with the following content: -.. code-block:: nim - + ```nim {.compile: "logic.c".} proc addTwoIntegers(a, b: cint): cint {.importc.} when isMainModule: echo addTwoIntegers(3, 7) + ``` -With these two files in place, you can run ``nim c -r calculator.nim`` and +With these two files in place, you can run `nim c -r calculator.nim`:cmd: and the Nim compiler will compile the ``logic.c`` file in addition to -``calculator.nim`` and link both into an executable, which outputs ``10`` when +``calculator.nim`` and link both into an executable, which outputs `10` when run. Another way to link the C file statically and get the same effect would -be remove the line with the ``compile`` pragma and run the following typical -Unix commands:: +be to remove the line with the `compile` pragma and run the following +typical Unix commands: - $ gcc -c logic.c - $ ar rvs mylib.a logic.o - $ nim c --passL:mylib.a -r calculator.nim + ```cmd + gcc -c logic.c + ar rvs mylib.a logic.o + nim c --passL:mylib.a -r calculator.nim + ``` Just like in this example we pass the path to the ``mylib.a`` library (and we could as well pass ``logic.o``) we could be passing switches to link any other static C library. -JavaScript invocation example -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### JavaScript invocation example Create a ``host.html`` file with the following content: -.. code-block:: - + ``` <html><body> <script type="text/javascript"> function addTwoIntegers(a, b) @@ -195,69 +209,70 @@ Create a ``host.html`` file with the following content: </script> <script type="text/javascript" src="calculator.js"></script> </body></html> + ``` Create a ``calculator.nim`` file with the following content (or reuse the one from the previous section): -.. code-block:: nim - + ```nim proc addTwoIntegers(a, b: int): int {.importc.} when isMainModule: echo addTwoIntegers(3, 7) + ``` -Compile the Nim code to JavaScript with ``nim js -o:calculator.js -calculator.nim`` and open ``host.html`` in a browser. If the browser supports -javascript, you should see the value ``10`` in the browser's console. Use the -`dom module <dom.html>`_ for specific DOM querying and modification procs -or take a look at `karax <https://github.com/pragmagic/karax>`_ for how to -develop browser based applications. +Compile the Nim code to JavaScript with `nim js -o:calculator.js +calculator.nim`:cmd: and open ``host.html`` in a browser. If the browser supports +javascript, you should see the value `10` in the browser's console. Use the +[dom module](dom.html) for specific DOM querying and modification procs +or take a look at [karax](https://github.com/pragmagic/karax) for how to +develop browser-based applications. Backend code calling Nim ------------------------ -Backend code can interface with Nim code exposed through the `exportc -pragma <manual.html#foreign-function-interface-exportc-pragma>`_. The -``exportc`` pragma is the *generic* way of making Nim symbols available to -the backends. By default the Nim compiler will mangle all the Nim symbols to -avoid any name collision, so the most significant thing the ``exportc`` pragma +Backend code can interface with Nim code exposed through the [exportc +pragma](manual.html#foreign-function-interface-exportc-pragma). The +`exportc` pragma is the *generic* way of making Nim symbols available to +the backends. By default, the Nim compiler will mangle all the Nim symbols to +avoid any name collision, so the most significant thing the `exportc` pragma does is maintain the Nim symbol name, or if specified, use an alternative symbol for the backend in case the symbol rules don't match. The JavaScript target doesn't have any further interfacing considerations since it also has garbage collection, but the C targets require you to -initialize Nim's internals, which is done calling a ``NimMain`` function. +initialize Nim's internals, which is done calling a `NimMain` function. Also, C code requires you to specify a forward declaration for functions or the compiler will assume certain types for the return value and parameters which will likely make your program crash at runtime. -The Nim compiler can generate a C interface header through the ``--header`` -command line switch. The generated header will contain all the exported -symbols and the ``NimMain`` proc which you need to call before any other -Nim code. +The name `NimMain` can be influenced via the `--nimMainPrefix:prefix` switch. +Use `--nimMainPrefix:MyLib` and the function to call is named `MyLibNimMain`. +When compiling to static or dynamic libraries, they don't call destructors of global variables as normal Nim programs would do. A C API `NimDestroyGlobals` is provided to call these global destructors. -Nim invocation example from C -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Create a ``fib.nim`` file with the following content: +### Nim invocation example from C -.. code-block:: nim +Create a ``fib.nim`` file with the following content: + ```nim proc fib(a: cint): cint {.exportc.} = if a <= 2: result = 1 else: result = fib(a - 1) + fib(a - 2) + ``` Create a ``maths.c`` file with the following content: -.. code-block:: c - - #include "fib.h" + ```c #include <stdio.h> + int fib(int a); + void NimMain(); + int main(void) { NimMain(); @@ -265,65 +280,68 @@ Create a ``maths.c`` file with the following content: printf("Fib of %d is %d\n", f, fib(f)); return 0; } + ``` Now you can run the following Unix like commands to first generate C sources from the Nim code, then link them into a static binary along your main C -program:: +program: - $ nim c --noMain --noLinking --header:fib.h fib.nim - $ gcc -o m -I$HOME/.cache/nim/fib_d -Ipath/to/nim/lib $HOME/.cache/nim/fib_d/*.c maths.c + ```cmd + nim c --noMain --noLinking fib.nim + gcc -o m -I$HOME/.cache/nim/fib_d -Ipath/to/nim/lib $HOME/.cache/nim/fib_d/*.c maths.c + ``` The first command runs the Nim compiler with three special options to avoid -generating a ``main()`` function in the generated files, avoid linking the -object files into a final binary, and explicitly generate a header file for C -integration. All the generated files are placed into the ``nimcache`` +generating a `main()`:c: function in the generated files and to avoid linking the +object files into a final binary. All the generated files are placed into the ``nimcache`` directory. That's why the next command compiles the ``maths.c`` source plus all the ``.c`` files from ``nimcache``. In addition to this path, you also have to tell the C compiler where to find Nim's ``nimbase.h`` header file. Instead of depending on the generation of the individual ``.c`` files you can -also ask the Nim compiler to generate a statically linked library:: +also ask the Nim compiler to generate a statically linked library: - $ nim c --app:staticLib --noMain --header fib.nim - $ gcc -o m -Inimcache -Ipath/to/nim/lib libfib.nim.a maths.c + ```cmd + nim c --app:staticLib fib.nim + gcc -o m -Inimcache -Ipath/to/nim/lib maths.c libfib.nim.a + ``` The Nim compiler will handle linking the source files generated in the ``nimcache`` directory into the ``libfib.nim.a`` static library, which you can -then link into your C program. Note that these commands are generic and will +then link into your C program. Note that these commands are generic and will vary for each system. For instance, on Linux systems you will likely need to -use ``-ldl`` too to link in required dlopen functionality. +use `-ldl`:option: too to link in required dlopen functionality. -Nim invocation example from JavaScript -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Nim invocation example from JavaScript Create a ``mhost.html`` file with the following content: -.. code-block:: - + ``` <html><body> <script type="text/javascript" src="fib.js"></script> <script type="text/javascript"> alert("Fib for 9 is " + fib(9)); </script> </body></html> + ``` Create a ``fib.nim`` file with the following content (or reuse the one from the previous section): -.. code-block:: nim - + ```nim proc fib(a: cint): cint {.exportc.} = if a <= 2: result = 1 else: result = fib(a - 1) + fib(a - 2) + ``` -Compile the Nim code to JavaScript with ``nim js -o:fib.js fib.nim`` and +Compile the Nim code to JavaScript with `nim js -o:fib.js fib.nim`:cmd: and open ``mhost.html`` in a browser. If the browser supports javascript, you should see an alert box displaying the text ``Fib for 9 is 34``. As mentioned -earlier, JavaScript doesn't require an initialisation call to ``NimMain`` or -similar function and you can call the exported Nim proc directly. +earlier, JavaScript doesn't require an initialization call to `NimMain` or +a similar function and you can call the exported Nim proc directly. Nimcache naming logic @@ -332,16 +350,16 @@ Nimcache naming logic The `nimcache`:idx: directory is generated during compilation and will hold either temporary or final files depending on your backend target. The default name for the directory depends on the used backend and on your OS but you can -use the ``--nimcache`` `compiler switch -<nimc.html#compiler-usage-command-line-switches>`_ to change it. +use the `--nimcache`:option: [compiler switch]( +nimc.html#compiler-usage-commandminusline-switches) to change it. Memory management ================= -In the previous sections the ``NimMain()`` function reared its head. Since +In the previous sections, the `NimMain()` function reared its head. Since JavaScript already provides automatic memory management, you can freely pass -objects between the two language without problems. In C and derivate languages +objects between the two languages without problems. In C and derivate languages you need to be careful about what you do and how you share memory. The previous examples only dealt with simple scalar values, but passing a Nim string to C, or reading back a C string in Nim already requires you to be @@ -351,32 +369,22 @@ aware of who controls what to avoid crashing. Strings and C strings --------------------- -The manual mentions that `Nim strings are implicitly convertible to -cstrings <manual.html#types-cstring-type>`_ which makes interaction usually +The manual mentions that [Nim strings are implicitly convertible to +cstrings](manual.html#types-cstring-type) which makes interaction usually painless. Most C functions accepting a Nim string converted to a -``cstring`` will likely not need to keep this string around and by the time -they return the string won't be needed any more. However, for the rare cases -where a Nim string has to be preserved and made available to the C backend -as a ``cstring``, you will need to manually prevent the string data from being -freed with `GC_ref <system.html#GC_ref,string>`_ and `GC_unref -<system.html#GC_unref,string>`_. +`cstring` will likely not need to keep this string around and by the time +they return the string won't be needed anymore. A similar thing happens with C code invoking Nim code which returns a -``cstring``. Consider the following proc: - -.. code-block:: nim +`cstring`. Consider the following proc: + ```nim proc gimme(): cstring {.exportc.} = result = "Hey there C code! " & $rand(100) + ``` -Since Nim's garbage collector is not aware of the C code, once the -``gimme`` proc has finished it can reclaim the memory of the ``cstring``. -However, from a practical standpoint, the C code invoking the ``gimme`` -function directly will be able to use it since Nim's garbage collector has -not had a chance to run *yet*. This gives you enough time to make a copy for -the C side of the program, as calling any further Nim procs *might* trigger -garbage collection making the previously returned string garbage. Or maybe you -are `yourself triggering the collection <gc.html>`_. +Since Nim's reference counting mechanism is not aware of the C code, once the +`gimme` proc has finished it can reclaim the memory of the `cstring`. Custom data types @@ -384,44 +392,15 @@ Custom data types Just like strings, custom data types that are to be shared between Nim and the backend will need careful consideration of who controls who. If you want -to hand a Nim reference to C code, you will need to use `GC_ref -<system.html#GC_ref,ref.T>`_ to mark the reference as used, so it does not get -freed. And for the C backend you will need to expose the `GC_unref -<system.html#GC_unref,ref.T>`_ proc to clean up this memory when it is not -required any more. +to hand a Nim reference to C code, you will need to use [GC_ref]( +system.html#GC_ref,ref.T) to mark the reference as used, so it does not get +freed. And for the C backend you will need to expose the [GC_unref]( +system.html#GC_unref,ref.T) proc to clean up this memory when it is not +required anymore. Again, if you are wrapping a library which *mallocs* and *frees* data structures, you need to expose the appropriate *free* function to Nim so you can clean it up. And of course, once cleaned you should avoid accessing it from Nim (or C for that matter). Typically C data structures have their own -``malloc_structure`` and ``free_structure`` specific functions, so wrapping +`malloc_structure`:c: and `free_structure`:c: specific functions, so wrapping these for the Nim side should be enough. - - -Thread coordination -------------------- - -When the ``NimMain()`` function is called Nim initializes the garbage -collector to the current thread, which is usually the main thread of your -application. If your C code later spawns a different thread and calls Nim -code, the garbage collector will fail to work properly and you will crash. - -As long as you don't use the threadvar emulation Nim uses native thread -variables, of which you get a fresh version whenever you create a thread. You -can then attach a GC to this thread via - -.. code-block:: nim - - system.setupForeignThreadGc() - -It is **not** safe to disable the garbage collector and enable it after the -call from your background thread even if the code you are calling is short -lived. - -Before the thread exits, you should tear down the thread's GC to prevent memory -leaks by calling - -.. code-block:: nim - - system.tearDownForeignThreadGc() - diff --git a/doc/basicopt.txt b/doc/basicopt.txt index cdd8ab9be..e8133d227 100644 --- a/doc/basicopt.txt +++ b/doc/basicopt.txt @@ -1,13 +1,12 @@ -:: nim command [options] [projectfile] [arguments] Command: - //compile, c compile project with default code generator (C) - //r compile to $nimcache/projname, run with [arguments] - using backend specified by `--backend` (default: c) - //doc generate the documentation for inputfile for - backend specified by `--backend` (default: c) + //compile, c compile project with default code generator (C) + //r compile to $nimcache/projname, run with `arguments` + using backend specified by `--backend` (default: c) + //doc generate the documentation for inputfile for + backend specified by `--backend` (default: c) Arguments: arguments are passed to the program being run (if --run option is selected) @@ -27,10 +26,13 @@ Options: -a, --assertions:on|off turn assertions on|off --opt:none|speed|size optimize not at all or for speed|size Note: use -d:release for a release build! - --debugger:native Use native debugger (gdb) + --debugger:native use native debugger (gdb) --app:console|gui|lib|staticlib generate a console app|GUI app|DLL|static library -r, --run run the compiled program with given arguments + --eval:cmd evaluate nim code directly; e.g.: `nim --eval:"echo 1"` + defaults to `e` (nimscript) but customizable: + `nim r --eval:'for a in stdin.lines: echo a'` --fullhelp show all command line switches -h, --help show this help -v, --version show detailed version information diff --git a/doc/codeowners.rst b/doc/codeowners.rst deleted file mode 100644 index 4b4a5810e..000000000 --- a/doc/codeowners.rst +++ /dev/null @@ -1,65 +0,0 @@ -=========== -Code owners -=========== - - -Subsystems and code owners --------------------------- - -*Note*: This list is incomplete, in doubt dom96 is responsible for the standard -library, araq is responsible for the compiler. - - -Compiler -~~~~~~~~ - -=========================== ====================================================== -subsystem owner(s) -=========================== ====================================================== -Parsing, Lexing araq -Renderer cooldome, araq -Order of passes cooldome -Semantic Checking araq -Virtual machine jangko, GULPF, araq -Sempass2: effects tracking cooldome, araq -type system, concepts zahary -transf cooldome, araq -semfold constant folding araq -template instantiation zahary, araq -term rewriting macros cooldome, araq -closure iterators yglukhov, araq -lambda lifting yglukhov, araq -c, cpp codegen lemonboy, araq -js codegen yglukhov, lemonboy -alias analysis araq -dfa, writetracking araq -parallel, multithreading araq -incremental araq -sizeof computations krux02 -Exception handling cooldome, araq -=========================== ====================================================== - - - -Standard library -~~~~~~~~~~~~~~~~ - -====================== ====================================================== -subsystem owner(s) -====================== ====================================================== -async dom96 -strutils araq -sequtils dom96, araq -times GULPF -os dom96, araq -re araq -nre flaviu -math, fenv krux02, cooldome -io dom96 -garbage collector araq -Go garbage collector stefantalpalaru -coroutines Rokas Kupstys -collections GULPF -parseopt araq -json dom96 -====================== ====================================================== diff --git a/doc/contributing.md b/doc/contributing.md new file mode 100644 index 000000000..420c1438e --- /dev/null +++ b/doc/contributing.md @@ -0,0 +1,798 @@ +============ +Contributing +============ + +.. default-role:: code +.. include:: rstcommon.rst + +.. contents:: + + +Contributing happens via "Pull requests" (PR) on GitHub. Every PR needs to be +reviewed before it can be merged and the Continuous Integration should be green. +The title of a PR should contain a brief description. If it fixes an issue, +in addition to the number of the issue, the title should also contain a description +of the issue. + +The PR has to be approved by two core developers or by Araq. + + + +Writing tests +============= + +There are 4 types of tests: + +1. `runnableExamples` documentation comment tests, ran by `nim doc mymod.nim`:cmd: + These end up in documentation and ensure documentation stays in sync with code. + +2. separate test files, e.g.: ``tests/stdlib/tos.nim``. + In nim repo, `testament`:cmd: (see below) runs all + ``$nim/tests/*/t*.nim`` test files; + for nimble packages, see https://github.com/nim-lang/nimble#tests. + +3. (deprecated) tests in `when isMainModule:` block, ran by `nim r mymod.nim`:cmd:. + `nimble test`:cmd: can run those in nimble packages when specified in a + `task "test"`. + +4. (not preferred) ``.. code-block:: nim`` RST snippets; + these should only be used in rst sources, + in nim sources `runnableExamples` should now always be preferred to those for + several reasons (cleaner syntax, syntax highlights, batched testing, and + parameter `rdoccmd` allows customization). + +Not all the tests follow the convention here, feel free to change the ones +that don't. Always leave the code cleaner than you found it. + +Stdlib +------ + +Each stdlib module (anything under ``lib/``, e.g. ``lib/pure/os.nim``) should +preferably have a corresponding separate test file, e.g. ``tests/stdlib/tos.nim``. +The old convention was to add a `when isMainModule:` block in the source file, +which only gets executed when the tester is building the file. + +Each test should be in a separate `block:` statement, such that +each has its own scope. Use boolean conditions and `doAssert` for the +testing by itself, don't rely on echo statements or similar; in particular, avoid +things like `echo "done"`. Don't use `unittest.suite` and `unittest.test`. + +Sample test: + + ```nim + block: # foo + doAssert foo(1) == 10 + + block: # bug #1234 + static: doAssert 1+1 == 2 + + block: # bug #1235 + var seq2D = newSeqWith(4, newSeq[bool](2)) + seq2D[0][0] = true + seq2D[1][0] = true + seq2D[0][1] = true + doAssert seq2D == @[@[true, true], @[true, false], + @[false, false], @[false, false]] + # doAssert with `not` can now be done as follows: + doAssert not (1 == 2) + ``` + +Always refer to a GitHub issue using the following exact syntax: ``bug #1234`` as shown +above, so that it's consistent and easier to search or for tooling. Some browser +extensions (e.g. https://github.com/sindresorhus/refined-github) will even turn those +in clickable links when it works. + +Rationale for using a separate test file instead of `when isMainModule:` block: +* allows custom compiler flags or testing options (see details below) +* faster CI since they can be joined in ``megatest`` (combined into a single test) +* avoids making the parser do un-necessary work when a source file is merely imported +* avoids mixing source and test code when reporting line of code statistics or code coverage + +Compiler +-------- + +The tests for the compiler use a testing tool called `testament`:cmd:. They are all +located in ``tests/`` (e.g.: ``tests/destructor/tdestructor3.nim``). +Each test has its own file. All test files are prefixed with `t`. If you want +to create a file for import into another test only, use the prefix `m`. + +At the beginning of every test is the expected behavior of the test. +Possible keys are: + +- `cmd`: A compilation command template e.g. `nim $target --threads:on $options $file`:cmd: +- `output`: The expected output (stdout + stderr), most likely via `echo` +- `exitcode`: Exit code of the test (via `exit(number)`) +- `errormsg`: The expected compiler error message +- `file`: The file the errormsg was produced at +- `line`: The line the errormsg was produced at + +For a full spec, see here: ``testament/specs.nim`` + +An example of a test: + + ```nim + discard """ + errormsg: "type mismatch: got (PTest)" + """ + + type + PTest = ref object + + proc test(x: PTest, y: int) = nil + + var buf: PTest + buf.test() + ``` + + +Running tests +============= + +You can run the tests with + + ```cmd + ./koch tests + ``` + +which will run a good subset of tests. Some tests may fail. If you +only want to see the output of failing tests, go for + + ```cmd + ./koch tests --failing all + ``` + +You can also run only a single category of tests. A category is a subdirectory +in the ``tests/`` directory. There are a couple of special categories; for a +list of these, see ``testament/categories.nim``, at the bottom. + + ```cmd + ./koch tests c lib # compiles / runs stdlib modules, including `isMainModule` tests + ./koch tests c megatest # runs a set of tests that can be combined into 1 + ``` + +To run a single test: + + ```cmd + ./koch test run <category>/<name> # e.g.: tuples/ttuples_issues + ./koch test run tests/stdlib/tos.nim # can also provide relative path + ``` + +For reproducible tests (to reproduce an environment more similar to the one +run by Continuous Integration on GitHub actions/azure pipelines), you may want to disable your +local configuration (e.g. in ``~/.config/nim/nim.cfg``) which may affect some +tests; this can also be achieved by using +`export XDG_CONFIG_HOME=pathtoAlternateConfig`:cmd: before running `./koch`:cmd: +commands. + + +Comparing tests +=============== + +Test failures can be grepped using ``Failure:``. + +The tester can compare two test runs. First, you need to create a +reference test. You'll also need to the commit id, because that's what +the tester needs to know in order to compare the two. + + ```cmd + git checkout devel + DEVEL_COMMIT=$(git rev-parse HEAD) + ./koch tests + ``` + +Then switch over to your changes and run the tester again. + + ```cmd + git checkout your-changes + ./koch tests + ``` + +Then you can ask the tester to create a ``testresults.html`` which will +tell you if any new tests passed/failed. + + ```cmd + ./koch tests --print html $DEVEL_COMMIT + ``` + + +Deprecation +=========== + +Backwards compatibility is important. When renaming types, procedures, etc. the old name +must be marked as deprecated using the `deprecated` pragma: + + ```nim + # for routines (proc/template/macro/iterator) and types: + proc oldProc(a: int, b: float): bool {.deprecated: + "deprecated since v1.2.3; use `newImpl: string -> int` instead".} = discard + + # for (const/var/let/fields) the msg is not yet supported: + const Foo {.deprecated.} = 1 + + # for enum types, you can deprecate the type or some elements + # (likewise with object types and their fields): + type Bar {.deprecated.} = enum bar0, bar1 + type Barz = enum baz0, baz1 {.deprecated.}, baz2 + ``` + + +See also [Deprecated](manual.html#pragmas-deprecated-pragma) +pragma in the manual. + + +Documentation +============= + +When contributing new procs, be sure to add documentation, especially if +the proc is public. Even private procs benefit from documentation and can be +viewed using `nim doc --docInternal foo.nim`:cmd:. +Documentation begins on the line +following the `proc` definition, and is prefixed by `##` on each line. + +Runnable code examples are also encouraged, to show typical behavior with a few +test cases (typically 1 to 3 `assert` statements, depending on complexity). +These `runnableExamples` are automatically run by `nim doc mymodule.nim`:cmd: +as well as `testament`:cmd: and guarantee they stay in sync. + + ```nim + proc addBar*(a: string): string = + ## Adds "Bar" to `a`. + runnableExamples: + assert "baz".addBar == "bazBar" + result = a & "Bar" + ``` + +See [parentDir](os.html#parentDir,string) example. + +The RestructuredText Nim uses has a special syntax for including code snippets +embedded in documentation; these are not run by `nim doc`:cmd: and therefore are +not guaranteed to stay in sync, so `runnableExamples` is almost always preferred: + + ````nim + proc someProc*(): string = + ## Returns "something" + ## + ## ``` + ## echo someProc() # "something" + ## ``` + result = "something" # single-hash comments do not produce documentation + ```` + +The \`\`\` followed by a newline and an indentation instructs the +`nim doc`:cmd: command to produce syntax-highlighted example code with the +documentation (\`\`\` is sufficient inside a ``.nim`` module, while from +a ``.md`` one needs to set the language explicitly as \`\`\`nim). + +When forward declaration is used, the documentation should be included with the +first appearance of the proc. + + ```nim + proc hello*(): string + ## Put documentation here + proc nothing() = discard + proc hello*(): string = + ## ignore this + echo "hello" + ``` + +The preferred documentation style is to begin with a capital letter and use +the third-person singular. That is, between: + + ```nim + proc hello*(): string = + ## Returns "hello" + result = "hello" + ``` + +or + + ```nim + proc hello*(): string = + ## say hello + result = "hello" + ``` + +the first is preferred. + +When you specify an *RST role* (highlighting/interpretation marker) do it +in the postfix form for uniformity, that is after \`text in backticks\`. +For example an ``:idx:`` role for referencing a topic ("SQLite" in the +example below) from [Nim Index] can be used in doc comment this way: + + ```nim + ## A higher level `SQLite`:idx: database wrapper. + ``` + +.. _`Nim Index`: https://nim-lang.org/docs/theindex.html + +Inline monospaced text can be input using \`single backticks\` or +\`\`double backticks\`\`. The former are syntactically highlighted, +the latter are not. +To avoid accidental highlighting follow this rule in ``*.nim`` files: + +* Use single backticks for fragments of code in Nim and other + programming languages, including identifiers, in ``*.nim`` files. + + For languages other than Nim add a role after final backtick, + e.g. for C++ inline highlighting: + + `#include <stdio.h>`:cpp: + + For a currently unsupported language add the `:code:` role, + like for SQL in this example: + + `SELECT * FROM <table_name>;`:code: + + Highlight shell commands by ``:cmd:`` role; for command line options use + ``:option:`` role, e.g.: \`--docInternal\`:option:. + +* Use double backticks: + + * For file names: \`\`os.nim\`\` + * For fragments of strings **not** enclosed by `"` and `"` and not + related to code, e.g. text of compiler messages + * When code ends with a standalone ``\`` (otherwise a combination of + ``\`` and a final \` would get escaped) + +.. Note:: ``*.rst`` files have ``:literal:`` as their default role. + So for them the rule above is only applicable if the ``:nim:`` role + is set up manually as the default [^1]: + + .. role:: nim(code) + :language: nim + .. default-role:: nim + + The first 2 lines are for other RST implementations, + including Github one. + + [^1]: this is fulfilled when ``doc/rstcommon.rst`` is included. + +Best practices +============== + +Note: these are general guidelines, not hard rules; there are always exceptions. +Code reviews can just point to a specific section here to save time and +propagate best practices. + +.. _define_needs_prefix: +New `defined(foo)` symbols need to be prefixed by the nimble package name, or +by `nim` for symbols in nim sources (e.g. compiler, standard library). This is +to avoid name conflicts across packages. + + ```nim + # if in nim sources + when defined(allocStats): discard # bad, can cause conflicts + when defined(nimAllocStats): discard # preferred + # if in a package `cligen`: + when defined(debug): discard # bad, can cause conflicts + when defined(cligenDebug): discard # preferred + ``` + +.. _noimplicitbool: +Take advantage of no implicit bool conversion + + ```nim + doAssert isValid() == true + doAssert isValid() # preferred + ``` + +.. _design_for_mcs: +Design with method call syntax chaining in mind + + ```nim + proc foo(cond: bool, lines: seq[string]) # bad + proc foo(lines: seq[string], cond: bool) # preferred + # can be called as: `getLines().foo(false)` + ``` + +.. _avoid_quit: +Use exceptions (including `assert` / `doAssert`) instead of `quit` +rationale: https://forum.nim-lang.org/t/4089 + + ```nim + quit() # bad in almost all cases + doAssert() # preferred + ``` + +.. _tests_use_doAssert: +Use `doAssert` (or `unittest.check`, `unittest.require`), not `assert` in all +tests, so they'll be enabled even with `--assertions:off`:option:. + + ```nim + block: # foo + assert foo() # bad + doAssert foo() # preferred + ``` + +.. _runnableExamples_use_assert: +An exception to the above rule is `runnableExamples` and ``code-block`` rst blocks +intended to be used as `runnableExamples`, which for brevity use `assert` +instead of `doAssert`. Note that `nim doc -d:danger main`:cmd: won't pass `-d:danger`:option: to the +`runnableExamples`, but `nim doc --doccmd:-d:danger main`:cmd: would, and so would the +second example below: + + ```nim + runnableExamples: + doAssert foo() # bad + assert foo() # preferred + + runnableExamples("-d:danger"): + doAssert foo() # `assert` would be disabled here, so `doAssert` makes more sense + ``` + +.. _delegate_printing: +Delegate printing to caller: return `string` instead of calling `echo` +rationale: it's more flexible (e.g. allows the caller to call custom printing, +including prepending location info, writing to log files, etc.). + + ```nim + proc foo() = echo "bar" # bad + proc foo(): string = "bar" # preferred (usually) + ``` + +.. _use_Option: +(Ongoing debate) Consider using Option instead of return bool + var argument, +unless stack allocation is needed (e.g. for efficiency). + + ```nim + proc foo(a: var Bar): bool + proc foo(): Option[Bar] + ``` + +.. _use_doAssert_not_echo: +Tests (including in testament) should always prefer assertions over `echo`, +except when that's not possible. It's more precise, easier for readers and +maintainers to where expected values refer to. See for example +https://github.com/nim-lang/Nim/pull/9335 and https://forum.nim-lang.org/t/4089 + + ```nim + echo foo() # adds a line for testament in `output:` block inside `discard`. + doAssert foo() == [1, 2] # preferred, except when not possible to do so. + ``` + + +The `git`:cmd: stuff +==================== + +General commit rules +-------------------- + +1. Important, critical bugfixes that have a tiny chance of breaking + somebody's code should be backported to the latest stable release + branch (currently 1.4.x) and maybe also all the way back to the 1.0.x branch. + The commit message should contain the tag ``[backport]`` for "backport to the latest + stable release" and the tag ``[backport:$VERSION]`` for backporting back to the + given $VERSION (and all newer releases). + +2. If you introduce changes which affect backward compatibility, + make breaking changes, or have PR which is tagged as ``[feature]``, + the changes should be mentioned in [the changelog]( + https://github.com/nim-lang/Nim/blob/devel/changelog.md). + +3. All changes introduced by the commit (diff lines) must be related to the + subject of the commit. + + If you change something unrelated to the subject parts of the file, because + your editor reformatted automatically the code or whatever different reason, + this should be excluded from the commit. + + *Tip:* Never commit everything as-is using `git commit -a`:cmd:, but review + carefully your changes with `git add -p`:cmd:. + +4. Changes should not introduce any trailing whitespace. + + Always check your changes for whitespace errors using `git diff --check`:cmd: + or add the following ``pre-commit`` hook: + + ```cmd + #!/bin/sh + git diff --check --cached || exit $? + ``` +5. Describe your commit and use your common sense. + Example commit message: + + Fixes #123; refs #124 + + indicates that issue ``#123`` is completely fixed (GitHub may automatically + close it when the PR is committed), whereas issue ``#124`` is referenced + (e.g.: partially fixed) and won't close the issue when committed. + +6. PR body (not just PR title) should contain references to fixed/referenced GitHub + issues, e.g.: ``fix #123`` or ``refs #123``. This is so that you get proper + cross-referencing from linked issue to the PR (GitHub won't make those links + with just a PR title, and commit messages aren't always sufficient to ensure + that, e.g. can't be changed after a PR is merged). + +7. Commits should be always be rebased against devel (so a fast-forward + merge can happen) + + e.g.: use `git pull --rebase origin devel`:cmd:. This is to avoid messing up + git history. + Exceptions should be very rare: when rebase gives too many conflicts, simply + squash all commits using the script shown in + https://github.com/nim-lang/Nim/pull/9356 + +8. Do not mix pure formatting changes (e.g. whitespace changes, nimpretty) or + automated changes with other code changes: these should be in + separate commits (and the merge on GitHub should not squash these into 1). + + +Continuous Integration (CI) +--------------------------- + +1. Continuous Integration is by default run on every push in a PR; this clogs + the CI pipeline and affects other PR's; if you don't need it (e.g. for WIP or + documentation only changes), add ``[skip ci]`` to your commit message title. + This convention is supported by our GitHub actions pipelines and our azure pipeline + (using custom logic, which should complete in < 1mn) as well as our former other pipelines: + [Appveyor]( + https://www.appveyor.com/docs/how-to/filtering-commits/#skip-directive-in-commit-message) + and [Travis]( + https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build). + +2. Consider enabling CI (azure, GitHub actions and builds.sr.ht) in your own Nim fork, and + waiting for CI to be green in that fork (fixing bugs as needed) before + opening your PR in the original Nim repo, to reduce CI congestion. Same + applies for updates on a PR: you can test commits on a separate private + branch before updating the main PR. + +Debugging CI failures, flaky tests, etc +--------------------------------------- + +1. First check the CI logs and search for `FAIL` to find why CI failed; if the + failure seems related to your PR, try to fix the code instead of restarting CI. + +2. If CI failure seems unrelated to your PR, it could be caused by a flaky test. + File a bug for it if it isn't already reported. A PR push (or opening/closing PR) + will re-trigger all CI jobs (even successful ones, which can be wasteful). Instead, + request collaboration from the Nim team. The Nim team should + follow these instructions to only restart the jobs that failed: + + * Azure: if on your own fork, it's possible from inside azure console + (e.g. ``dev.azure.com/username/username/_build/results?buildId=1430&view=results``) via + ``rerun failed jobs`` on top. + If either on you own fork or in Nim repo, it's possible from inside GitHub UI + under checks tab, see https://github.com/timotheecour/Nim/issues/211#issuecomment-629751569 + * GitHub actions: under "Checks" tab, click "Re-run jobs" in the right. + * builds.sr.ht: create a SourceHut account so that you can restart a PR job as illustrated. + builds.sr.ht also allows you to ssh to a CI machine which can help a lot for debugging + issues, see docs in https://man.sr.ht/builds.sr.ht/build-ssh.md and + https://drewdevault.com/2019/08/19/Introducing-shell-access-for-builds.html; see + https://man.sr.ht/tutorials/set-up-account-and-git.md to generate and upload ssh keys. + + +Code reviews +------------ + +1. Whenever possible, use GitHub's new 'Suggested change' in code reviews, which + saves time explaining the change or applying it; see also + https://forum.nim-lang.org/t/4317 + +2. When reviewing large diffs that may involve code moving around, GitHub's interface + doesn't help much, as it doesn't highlight moves. Instead, you can use something + like this, see visual results [here]( + https://github.com/nim-lang/Nim/pull/10431#issuecomment-456968196): + + ```cmd + git fetch origin pull/10431/head && git checkout FETCH_HEAD + git diff --color-moved-ws=allow-indentation-change --color-moved=blocks HEAD^ + ``` + +3. In addition, you can view GitHub-like diffs locally to identify what was changed + within a code block using `diff-highlight`:cmd: or `diff-so-fancy`:cmd:, e.g.: + + # put this in ~/.gitconfig: + [core] + pager = "diff-so-fancy | less -R" # or: use: `diff-highlight` + + + +.. include:: docstyle.md + + +Evolving the stdlib +=================== + +As outlined in https://github.com/nim-lang/RFCs/issues/173 there are a couple +of guidelines about what should go into the stdlib, what should be added and +what eventually should be removed. + + +What the compiler itself needs must be part of the stdlib +--------------------------------------------------------- + +Maybe in the future the compiler itself can depend on Nimble packages but for +the time being, we strive to have zero dependencies in the compiler as the +compiler is the root of the bootstrapping process and is also used to build +Nimble. + + +Vocabulary types must be part of the stdlib +------------------------------------------- + +These are types most packages need to agree on for better interoperability, +for example `Option[T]`. This rule also covers the existing collections like +`Table`, `CountTable` etc. "Sorted" containers based on a tree-like data +structure are still missing and should be added. + +Time handling, especially the `Time` type are also covered by this rule. + + +Existing, battle-tested modules stay +------------------------------------ + +Reason: There is no benefit in moving them around just to fulfill some design +fashion as in "Nim's core MUST BE SMALL". If you don't like an existing module, +don't import it. If a compilation target (e.g. JS) cannot support a module, +document this limitation. + +This covers modules like `os`, `osproc`, `strscans`, `strutils`, +`strformat`, etc. + + +Syntactic helpers can start as experimental stdlib modules +---------------------------------------------------------- + +Reason: Generally speaking as external dependencies they are not exposed +to enough users so that we can see if the shortcuts provide enough benefit +or not. Many programmers avoid external dependencies, especially for +"tiny syntactic improvements". However, this is only true for really good +syntactic improvements that have the potential to clean up other parts of +the Nim library substantially. If in doubt, new stdlib modules should start +as external, successful Nimble packages. + + + +Other new stdlib modules do not start as stdlib modules +------------------------------------------------------- + +As we strive for higher quality everywhere, it's easier to adopt existing, +battle-tested modules eventually rather than creating modules from scratch. + + +Little additions are acceptable +------------------------------- + +As long as they are documented and tested well, adding little helpers +to existing modules is acceptable. For two reasons: + +1. It makes Nim easier to learn and use in the long run. + ("Why does sequtils lack a `countIt`? + Because version 1.0 happens to have lacked it? Silly...") +2. To encourage contributions. Contributors often start with PRs that + add simple things, then they stay and also fix bugs. Nim is an + open source project and lives from people's contributions and involvement. + Newly introduced issues have to be balanced against motivating new people. We know where + to find perfectly designed pieces of software that have no bugs -- these are the systems + that nobody uses. + +Conventions +----------- +1. New stdlib modules should go under ``Nim/lib/std/``. The rationale is to + require users to import via `import std/foo` instead of `import foo`, + which would cause potential conflicts with nimble packages. + Note that this still applies for new modules in existing logical + directories, e.g.: use ``lib/std/collections/foo.nim``, + not ``lib/pure/collections/foo.nim``. + +2. New module names should prefer plural form whenever possible, e.g.: + ``std/sums.nim`` instead of ``std/sum.nim``. In particular, this reduces + chances of conflicts between module name and the symbols it defines. + Furthermore, module names should use `snake_case` and not use capital + letters, which cause issues when going from an OS without case + sensitivity to an OS with it. + + +Breaking Changes +================ + +Introducing breaking changes, no matter how well-intentioned, +creates long-term problems for the community, in particular those looking to promote +reusable Nim code in libraries: In the Nim distribution, critical security and bugfixes, +language changes and community improvements are bundled in a single distribution - it is +difficult to make partial upgrades with only benign changes. When one library depends on +a legacy behavior, it can no longer be used together with another library that does not, +breaking all downstream applications - the standard library is unique in that it sits at +the root of **all** dependency trees. + +There is a big difference between compile-time breaking changes and run-time breaking +changes. + + +Run-time breaking changes +------------------------- + +Run-time breaking changes are to be avoided at almost all costs: Nim is used for +mission critical applications which depend on behaviours that +are not covered by the test suite. As such, it's important that changes to the +*stable* parts of the standard library are made avoiding changing the existing +behaviors, even when the test suite continues to pass. + +Examples of run-time breaking changes: + +- Raising exceptions of a new type, compared to what's currently being raised. + +- Adding unconstrained or poorly constrained generic procs or macros + ("hash now works for all `ref T`"): This may cause code to behave differently + depending only on which modules are imported - common examples include `==` and `hash`. + +- Changing behavior of existing functions like: + + * "Nim's path handling procs like `getXDir` now consistently lack the trailing slash" + * "Nim's strformat implementation is now more consistent with Python" + +Instead, write new code that explicitly announces the feature you think we announced but +didn't. For example, `strformat` does not say "it's compatible with Python", it +says "inspired by Python's f-strings". This new code can be submitted to the stdlib +and the old code can be deprecated or published as a Nimble package. + +Sometimes, a run-time breaking change is most desirable: For example, a string +representation of a floating point number that "roundtrips" is much better than +a string representation that doesn't. These run-time breaking changes must start in the +state "opt-in" via a new `-d:nimPreviewX` or command line flag and then should become +the new default later, in follow-up versions. This way users can track +regressions more easily. ("git bisect" is not an acceptable alternative, that's for +Nim compiler developers, not for Nim users.) + +Above all else, additive approaches that don't change existing behaviors +should be preferred. + + +Compile-time breaking changes +----------------------------- + +Compile-time breaking changes are usually easier to handle, but for large code bases +they can also involve a large amount of work and can hinder the adoption of a new +Nim release. +Additive approaches are to be preferred here as well. + +Examples of compile-time breaking changes include (but are not limited to): + +* Renaming functions and modules, or moving things. Instead of a direct rename, + deprecate the old name and introduce a new one. +* Renaming the parameter names: Thanks to Nim's "named parameter" calling syntax + like `f(x = 0, y = 1)` this is a breaking change. Instead, live with the existing + parameter names. +* Adding an enum value to an existing enum. Nim's exhaustive case statements stop + compiling after such a change. Instead, consider to introduce new `bool` + fields/parameters. This can be impractical though, so we use good judgement + and our list of "important packages" to see if it doesn't break too much code + out there in practice. +* Adding a new proc to an existing stdlib module. However, for aesthetic reasons + this is often preferred over introducing a new module with just a single proc + inside. We use good judgement and our list of "important packages" to see if + it doesn't break too much code out there in practice. The new procs need to + be annotated with a `.since` annotation. + + +Compiler/language spec bugfixes +------------------------------- + +This can even be applied to compiler "bugfixes": If the compiler should have been +"pickier" in its handling of `typedesc`, instead of "fixing typedesc handling bugs", +consider the following solution: + +- Spec out how `typedesc` should really work and also spec out the cases where it + should not be allowed! +- Deprecate `typedesc` and name the new metatype something new like `typeArg`. +- Implement the spec. + + +Non-breaking changes +-------------------- + +Examples of changes that are considered non-breaking (or acceptable breaking changes) include: + +* Creating a new module. +* Adding an overload to an already overloaded proc. +* Adding new default parameters to an existing proc. It is assumed that you do not + use Nim's stdlib procs's addresses (that you don't use them as first class entities). +* Changing the calling convention from `nimcall` to `inline` + (but first RFC https://github.com/nim-lang/RFCs/issues/396 needs to be implemented). +* Changing the behavior from "crashing" into some other, well documented result (including + raising a Defect, but not raising an exception that does not inherit from Defect). +* Adding new fields to an existing object. + +Nim's introspection facilities imply that strictly speaking almost every addition can +break somebody's code. It is impractical to care about these cases, a change that only +affects introspection is not considered to be a breaking change. diff --git a/doc/contributing.rst b/doc/contributing.rst deleted file mode 100644 index a4baaef05..000000000 --- a/doc/contributing.rst +++ /dev/null @@ -1,557 +0,0 @@ -============ -Contributing -============ - -.. contents:: - - -Contributing happens via "Pull requests" (PR) on github. Every PR needs to be -reviewed before it can be merged and the Continuous Integration should be green. - -The PR has to be approved (and is often merged too) by one "code owner", either -by the code owner who is responsible for the subsystem the PR belongs to or by -two core developers or by Araq. - -See `codeowners <codeowners.html>`_ for more details. - - -Writing tests -============= - -There are 4 types of tests: - -1. ``runnableExamples`` documentation comment tests, ran by ``nim doc mymod.nim`` - These end up in documentation and ensure documentation stays in sync with code. - -2. separate test files, e.g.: ``tests/stdlib/tos.nim``. - In nim repo, `testament` (see below) runs all `$nim/tests/*/t*.nim` test files; - for nimble packages, see https://github.com/nim-lang/nimble#tests. - -3. (deprecated) tests in ``when isMainModule:`` block, ran by ``nim r mymod.nim``. - ``nimble test`` can run those in nimble packages when specified in a - `task "test"`. - -4. (not preferred) `.. code-block:: nim` RST snippets; these should only be used in rst sources, - in nim sources `runnableExamples` should now always be preferred to those for - several reasons (cleaner syntax, syntax highlights, batched testing, and - `rdoccmd` allows customization). - -Not all the tests follow the convention here, feel free to change the ones -that don't. Always leave the code cleaner than you found it. - -Stdlib ------- - -Each stdlib module (anything under ``lib/``, e.g. ``lib/pure/os.nim``) should -preferably have a corresponding separate test file, eg `tests/stdlib/tos.nim`. -The old convention was to add a ``when isMainModule:`` block in the source file, -which only gets executed when the tester is building the file. - -Each test should be in a separate ``block:`` statement, such that -each has its own scope. Use boolean conditions and ``doAssert`` for the -testing by itself, don't rely on echo statements or similar; in particular avoid -things like `echo "done"`. - -Sample test: - -.. code-block:: nim - - block: # bug #1234 - static: doAssert 1+1 == 2 - - block: # bug #1235 - var seq2D = newSeqWith(4, newSeq[bool](2)) - seq2D[0][0] = true - seq2D[1][0] = true - seq2D[0][1] = true - doAssert seq2D == @[@[true, true], @[true, false], - @[false, false], @[false, false]] - # doAssert with `not` can now be done as follows: - doAssert not (1 == 2) - -Always refer to a github issue using the following exact syntax: `bug #1234` as shown -above, so that it's consistent and easier to search or for tooling. Some browser -extensions (eg https://github.com/sindresorhus/refined-github) will even turn those -in clickable links when it works. - -Rationale for using a separate test file instead of `when isMainModule:` block: -* allows custom compiler flags or testing options (see details below) -* faster CI since they can be joined in `megatest` (combined into a single test) -* avoids making the parser do un-necessary work when a source file is merely imported -* avoids mixing source and test code when reporting line of code statistics or code coverage - -Compiler --------- - -The tests for the compiler use a testing tool called ``testament``. They are all -located in ``tests/`` (e.g.: ``tests/destructor/tdestructor3.nim``). -Each test has its own file. All test files are prefixed with ``t``. If you want -to create a file for import into another test only, use the prefix ``m``. - -At the beginning of every test is the expected behavior of the test. -Possible keys are: - -- ``cmd``: A compilation command template e.g. ``nim $target --threads:on $options $file`` -- ``output``: The expected output (stdout + stderr), most likely via ``echo`` -- ``exitcode``: Exit code of the test (via ``exit(number)``) -- ``errormsg``: The expected compiler error message -- ``file``: The file the errormsg was produced at -- ``line``: The line the errormsg was produced at - -For a full spec, see here: ``testament/specs.nim`` - -An example for a test: - -.. code-block:: nim - - discard """ - errormsg: "type mismatch: got (PTest)" - """ - - type - PTest = ref object - - proc test(x: PTest, y: int) = nil - - var buf: PTest - buf.test() - - -Running tests -============= - -You can run the tests with - -:: - - ./koch tests - -which will run a good subset of tests. Some tests may fail. If you -only want to see the output of failing tests, go for - -:: - - ./koch tests --failing all - -You can also run only a single category of tests. A category is a subdirectory -in the ``tests`` directory. There are a couple of special categories; for a -list of these, see ``testament/categories.nim``, at the bottom. - -:: - - ./koch tests c lib # compiles/runs stdlib modules, including `isMainModule` tests - ./koch tests c megatest # runs a set of tests that can be combined into 1 - -To run a single test: - -:: - - ./koch test run <category>/<name> # e.g.: tuples/ttuples_issues - ./koch test run tests/stdlib/tos.nim # can also provide relative path - -For reproducible tests (to reproduce an environment more similar to the one -run by Continuous Integration on travis/appveyor), you may want to disable your -local configuration (e.g. in ``~/.config/nim/nim.cfg``) which may affect some -tests; this can also be achieved by using -``export XDG_CONFIG_HOME=pathtoAlternateConfig`` before running ``./koch`` -commands. - - -Comparing tests -=============== - -Test failures can be grepped using ``Failure:``. - -The tester can compare two test runs. First, you need to create the -reference test. You'll also need to the commit id, because that's what -the tester needs to know in order to compare the two. - -:: - - git checkout devel - DEVEL_COMMIT=$(git rev-parse HEAD) - ./koch tests - -Then switch over to your changes and run the tester again. - -:: - - git checkout your-changes - ./koch tests - -Then you can ask the tester to create a ``testresults.html`` which will -tell you if any new tests passed/failed. - -:: - - ./koch tests --print html $DEVEL_COMMIT - - -Deprecation -=========== - -Backward compatibility is important, so instead of a rename you need to deprecate -the old name and introduce a new name: - -.. code-block:: nim - - # for routines (proc/template/macro/iterator) and types: - proc oldProc(a: int, b: float): bool {.deprecated: - "deprecated since v1.2.3; use `newImpl: string -> int` instead".} = discard - - # for (const/var/let/fields) the msg is not yet supported: - const Foo {.deprecated.} = 1 - - # for enum types, you can deprecate the type or some elements - # (likewise with object types and their fields): - type Bar {.deprecated.} = enum bar0, bar1 - type Barz = enum baz0, baz1 {.deprecated.}, baz2 - - -See also `Deprecated <manual.html#pragmas-deprecated-pragma>`_ -pragma in the manual. - - -Documentation -============= - -When contributing new procs, be sure to add documentation, especially if -the proc is public. Even private procs benefit from documentation and can be -viewed using ``nim doc --docInternal foo.nim``. -Documentation begins on the line -following the ``proc`` definition, and is prefixed by ``##`` on each line. - -Runnable code examples are also encouraged, to show typical behavior with a few -test cases (typically 1 to 3 ``assert`` statements, depending on complexity). -These ``runnableExamples`` are automatically run by ``nim doc mymodule.nim`` -as well as ``testament`` and guarantee they stay in sync. - -.. code-block:: nim - proc addBar*(a: string): string = - ## Adds "Bar" to `a`. - runnableExamples: - assert "baz".addBar == "bazBar" - result = a & "Bar" - -See `parentDir <os.html#parentDir,string>`_ example. - -The RestructuredText Nim uses has a special syntax for including code snippets -embedded in documentation; these are not run by ``nim doc`` and therefore are -not guaranteed to stay in sync, so ``runnableExamples`` is usually preferred: - -.. code-block:: nim - - proc someproc*(): string = - ## Return "something" - ## - ## .. code-block:: - ## echo someproc() # "something" - result = "something" # single-hash comments do not produce documentation - -The ``.. code-block:: nim`` followed by a newline and an indentation instructs the -``nim doc`` command to produce syntax-highlighted example code with the -documentation (``.. code-block::`` is sufficient from inside a nim module). - -When forward declaration is used, the documentation should be included with the -first appearance of the proc. - -.. code-block:: nim - - proc hello*(): string - ## Put documentation here - proc nothing() = discard - proc hello*(): string = - ## ignore this - echo "hello" - -The preferred documentation style is to begin with a capital letter and use -the imperative (command) form. That is, between: - -.. code-block:: nim - - proc hello*(): string = - ## Return "hello" - result = "hello" - -or - -.. code-block:: nim - - proc hello*(): string = - ## says hello - result = "hello" - -the first is preferred. - - -Best practices -============== - -Note: these are general guidelines, not hard rules; there are always exceptions. -Code reviews can just point to a specific section here to save time and -propagate best practices. - -.. _define_needs_prefix: -New `defined(foo)` symbols need to be prefixed by the nimble package name, or -by `nim` for symbols in nim sources (e.g. compiler, standard library). This is -to avoid name conflicts across packages. - -.. code-block:: nim - - # if in nim sources - when defined(allocStats): discard # bad, can cause conflicts - when defined(nimAllocStats): discard # preferred - # if in a pacakge `cligen`: - when defined(debug): discard # bad, can cause conflicts - when defined(cligenDebug): discard # preferred - -.. _noimplicitbool: -Take advantage of no implicit bool conversion - -.. code-block:: nim - - doAssert isValid() == true - doAssert isValid() # preferred - -.. _design_for_mcs: -Design with method call syntax chaining in mind - -.. code-block:: nim - - proc foo(cond: bool, lines: seq[string]) # bad - proc foo(lines: seq[string], cond: bool) # preferred - # can be called as: `getLines().foo(false)` - -.. _avoid_quit: -Use exceptions (including assert / doAssert) instead of ``quit`` -rationale: https://forum.nim-lang.org/t/4089 - -.. code-block:: nim - - quit() # bad in almost all cases - doAssert() # preferred - -.. _tests_use_doAssert: -Use ``doAssert`` (or ``require``, etc), not ``assert`` in all tests so they'll -be enabled even in release mode (except for tests in ``runnableExamples`` blocks -which for which ``nim doc`` ignores ``-d:release``). - -.. code-block:: nim - - when isMainModule: - assert foo() # bad - doAssert foo() # preferred - -.. _delegate_printing: -Delegate printing to caller: return ``string`` instead of calling ``echo`` -rationale: it's more flexible (e.g. allows caller to call custom printing, -including prepending location info, writing to log files, etc). - -.. code-block:: nim - - proc foo() = echo "bar" # bad - proc foo(): string = "bar" # preferred (usually) - -.. _use_Option: -[Ongoing debate] Consider using Option instead of return bool + var argument, -unless stack allocation is needed (e.g. for efficiency). - -.. code-block:: nim - - proc foo(a: var Bar): bool - proc foo(): Option[Bar] - -.. _use_doAssert_not_echo: -Tests (including in testament) should always prefer assertions over ``echo``, -except when that's not possible. It's more precise, easier for readers and -maintaners to where expected values refer to. See for example -https://github.com/nim-lang/Nim/pull/9335 and https://forum.nim-lang.org/t/4089 - -.. code-block:: nim - - echo foo() # adds a line for testament in `output:` block inside `discard`. - doAssert foo() == [1, 2] # preferred, except when not possible to do so. - - -The Git stuff -============= - -General commit rules --------------------- - -1. Important, critical bugfixes that have a tiny chance of breaking - somebody's code should be backported to the latest stable release - branch (currently 1.2.x) and maybe also to the 1.0 branch. - The commit message should contain the tag ``[backport]`` for "backport to all - stable releases" and the tag ``[backport:$VERSION]`` for backporting to the - given $VERSION. - -2. If you introduce changes which affect backwards compatibility, - make breaking changes, or have PR which is tagged as ``[feature]``, - the changes should be mentioned in `the changelog - <https://github.com/nim-lang/Nim/blob/devel/changelog.md>`_. - -3. All changes introduced by the commit (diff lines) must be related to the - subject of the commit. - - If you change something unrelated to the subject parts of the file, because - your editor reformatted automatically the code or whatever different reason, - this should be excluded from the commit. - - *Tip:* Never commit everything as is using ``git commit -a``, but review - carefully your changes with ``git add -p``. - -4. Changes should not introduce any trailing whitespace. - - Always check your changes for whitespace errors using ``git diff --check`` - or add following ``pre-commit`` hook: - - .. code-block:: sh - - #!/bin/sh - git diff --check --cached || exit $? - -5. Describe your commit and use your common sense. - Example commit message: - - ``Fixes #123; refs #124`` - - indicates that issue ``#123`` is completely fixed (github may automatically - close it when the PR is committed), wheres issue ``#124`` is referenced - (e.g.: partially fixed) and won't close the issue when committed. - -6. Commits should be always be rebased against devel (so a fast forward - merge can happen) - - e.g.: use ``git pull --rebase origin devel``. This is to avoid messing up - git history. - Exceptions should be very rare: when rebase gives too many conflicts, simply - squash all commits using the script shown in - https://github.com/nim-lang/Nim/pull/9356 - -7. Do not mix pure formatting changes (e.g. whitespace changes, nimpretty) or - automated changes (e.g. nimfix) with other code changes: these should be in - separate commits (and the merge on github should not squash these into 1). - - -Continuous Integration (CI) ---------------------------- - -1. Continuous Integration is by default run on every push in a PR; this clogs - the CI pipeline and affects other PR's; if you don't need it (e.g. for WIP or - documentation only changes), add ``[ci skip]`` to your commit message title. - This convention is supported by `Appveyor - <https://www.appveyor.com/docs/how-to/filtering-commits/#skip-directive-in-commit-message>`_ - and `Travis <https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build>`_. - -2. Consider enabling CI (travis and appveyor) in your own Nim fork, and - waiting for CI to be green in that fork (fixing bugs as needed) before - opening your PR in original Nim repo, so as to reduce CI congestion. Same - applies for updates on a PR: you can test commits on a separate private - branch before updating the main PR. - -Code reviews ------------- - -1. Whenever possible, use github's new 'Suggested change' in code reviews, which - saves time explaining the change or applying it; see also - https://forum.nim-lang.org/t/4317 - -2. When reviewing large diffs that may involve code moving around, github's interface - doesn't help much as it doesn't highlight moves. Instead you can use something - like this, see visual results `here <https://github.com/nim-lang/Nim/pull/10431#issuecomment-456968196>`_: - - .. code-block:: sh - - git fetch origin pull/10431/head && git checkout FETCH_HEAD - git diff --color-moved-ws=allow-indentation-change --color-moved=blocks HEAD^ - -3. In addition, you can view github-like diffs locally to identify what was changed - within a code block using `diff-highlight` or `diff-so-fancy`, e.g.: - - .. code-block:: sh - - # put this in ~/.gitconfig: - [core] - pager = "diff-so-fancy | less -R" # or: use: `diff-highlight` - - - -.. include:: docstyle.rst - - -Evolving the stdlib -=================== - -As outlined in https://github.com/nim-lang/RFCs/issues/173 there are a couple -of guidelines about what should go into the stdlib, what should be added and -what eventually should be removed. - - -What the compiler itself needs must be part of the stdlib ---------------------------------------------------------- - -Maybe in the future the compiler itself can depend on Nimble packages but for -the time being, we strive to have zero dependencies in the compiler as the -compiler is the root of the bootstrapping process and is also used to build -Nimble. - - -Vocabulary types must be part of the stdlib -------------------------------------------- - -These are types most packages need to agree on for better interoperability, -for example ``Option[T]``. This rule also covers the existing collections like -``Table``, ``CountTable`` etc. "Sorted" containers based on a tree-like data -structure are still missing and should be added. - -Time handling, especially the ``Time`` type are also covered by this rule. - - -Existing, battle-tested modules stay ------------------------------------- - -Reason: There is no benefit in moving them around just to fullfill some design -fashion as in "Nim's core MUST BE SMALL". If you don't like an existing module, -don't import it. If a compilation target (e.g. JS) cannot support a module, -document this limitation. - -This covers modules like ``os``, ``osproc``, ``strscans``, ``strutils``, -``strformat``, etc. - - -Syntactic helpers can start as experimental stdlib modules ----------------------------------------------------------- - -Reason: Generally speaking as external dependencies they are not exposed -to enough users so that we can see if the shortcuts provide enough benefit -or not. Many programmers avoid external dependencies, even moreso for -"tiny syntactic improvements". However, this is only true for really good -syntactic improvements that have the potential to clean up other parts of -the Nim library substantially. If in doubt, new stdlib modules should start -as external, successful Nimble packages. - - - -Other new stdlib modules do not start as stdlib modules -------------------------------------------------------- - -As we strive for higher quality everywhere, it's easier to adopt existing, -battle-tested modules eventually rather than creating modules from scratch. - - -Little additions are acceptable -------------------------------- - -As long as they are documented and tested well, adding little helpers -to existing modules is acceptable. For two reasons: - -1. It makes Nim easier to learn and use in the long run. - ("Why does sequtils lack a ``countIt``? - Because version 1.0 happens to have lacked it? Silly...") -2. To encourage contributions. Contributors often start with PRs that - add simple things and then they stay and also fix bugs. Nim is an - open source project and lives from people's contributions and involvement. - Newly introduced issues have to be balanced against motivating new people. We know where - to find perfectly designed pieces of software that have no bugs -- these are the systems - that nobody uses. diff --git a/doc/destructors.md b/doc/destructors.md new file mode 100644 index 000000000..e192fd362 --- /dev/null +++ b/doc/destructors.md @@ -0,0 +1,801 @@ +================================== +Nim Destructors and Move Semantics +================================== + +:Authors: Andreas Rumpf +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + + +About this document +=================== + +This document describes the ARC/ORC Nim runtime which does +not use classical GC algorithms anymore but is based on destructors and +move semantics. The advantages are that Nim programs become +oblivious to the involved heap sizes and programs are easier to write to make +effective use of multi-core machines. As a nice bonus, files and sockets and +the like can be written not to require manual `close` calls anymore. + +This document aims to be a precise specification about how +move semantics and destructors work in Nim. + + +Motivating example +================== + +With the language mechanisms described here, a custom seq could be +written as: + + ```nim test + type + myseq*[T] = object + len, cap: int + data: ptr UncheckedArray[T] + + proc `=destroy`*[T](x: myseq[T]) = + if x.data != nil: + for i in 0..<x.len: `=destroy`(x.data[i]) + dealloc(x.data) + + proc `=wasMoved`*[T](x: var myseq[T]) = + x.data = nil + + proc `=trace`[T](x: var myseq[T]; env: pointer) = + # `=trace` allows the cycle collector `--mm:orc` + # to understand how to trace the object graph. + if x.data != nil: + for i in 0..<x.len: `=trace`(x.data[i], env) + + proc `=copy`*[T](a: var myseq[T]; b: myseq[T]) = + # do nothing for self-assignments: + if a.data == b.data: return + `=destroy`(a) + `=wasMoved`(a) + a.len = b.len + a.cap = b.cap + if b.data != nil: + a.data = cast[typeof(a.data)](alloc(a.cap * sizeof(T))) + for i in 0..<a.len: + a.data[i] = b.data[i] + + proc `=dup`*[T](a: myseq[T]): myseq[T] {.nodestroy.} = + # an optimized version of `=wasMoved(tmp); `=copy(tmp, src)` + # usually present if a custom `=copy` hook is overridden + result = myseq[T](len: a.len, cap: a.cap, data: nil) + if a.data != nil: + result.data = cast[typeof(result.data)](alloc(result.cap * sizeof(T))) + for i in 0..<result.len: + result.data[i] = `=dup`(a.data[i]) + + proc `=sink`*[T](a: var myseq[T]; b: myseq[T]) = + # move assignment, optional. + # Compiler is using `=destroy` and `copyMem` when not provided + `=destroy`(a) + a.len = b.len + a.cap = b.cap + a.data = b.data + + proc add*[T](x: var myseq[T]; y: sink T) = + if x.len >= x.cap: + x.cap = max(x.len + 1, x.cap * 2) + x.data = cast[typeof(x.data)](realloc(x.data, x.cap * sizeof(T))) + x.data[x.len] = y + inc x.len + + proc `[]`*[T](x: myseq[T]; i: Natural): lent T = + assert i < x.len + x.data[i] + + proc `[]=`*[T](x: var myseq[T]; i: Natural; y: sink T) = + assert i < x.len + x.data[i] = y + + proc createSeq*[T](elems: varargs[T]): myseq[T] = + result = myseq[T]( + len: elems.len, + cap: elems.len, + data: cast[typeof(result.data)](alloc(result.cap * sizeof(T)))) + for i in 0..<result.len: result.data[i] = elems[i] + + proc len*[T](x: myseq[T]): int {.inline.} = x.len + ``` + + +Lifetime-tracking hooks +======================= + +The memory management for Nim's standard `string` and `seq` types as +well as other standard collections is performed via so-called +"Lifetime-tracking hooks", which are particular [type bound operators]( +manual.html#procedures-type-bound-operators). + +There are 6 different hooks for each (generic or concrete) object type `T` (`T` can also be a +`distinct` type) that are called implicitly by the compiler. + +(Note: The word "hook" here does not imply any kind of dynamic binding +or runtime indirections, the implicit calls are statically bound and +potentially inlined.) + + +`=destroy` hook +--------------- + +A `=destroy` hook frees the object's associated memory and releases +other associated resources. Variables are destroyed via this hook when +they go out of scope or when the routine they were declared in is about +to return. + +A `=destroy` hook is allowed to have a parameter of a `var T` or `T` type. Taking a `var T` type is deprecated. The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=destroy`(x: T) + ``` + +The general pattern in `=destroy` looks like: + + ```nim + proc `=destroy`(x: T) = + # first check if 'x' was moved to somewhere else: + if x.field != nil: + freeResource(x.field) + ``` + +A `=destroy` is implicitly annotated with `.raises: []`; a destructor +should not raise exceptions. For backwards compatibility the compiler +produces a warning for a `=destroy` that does raise. + +A `=destroy` can explicitly list the exceptions it can raise, if any, +but this of little utility as a raising destructor is implementation defined +behavior. Later versions of the language specification might cover this case precisely. + + +`=wasMoved` hook +---------------- + +A `=wasMoved` hook sets the object to a state that signifies to the destructor there is nothing to destroy. + +The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=wasMoved`(x: var T) + ``` + +Usually some pointer field inside the object is set to `nil`: + + ```nim + proc `=wasMoved`(x: var T) = + x.field = nil + ``` + + +`=sink` hook +------------ + +A `=sink` hook moves an object around, the resources are stolen from the source +and passed to the destination. It is ensured that the source's destructor does +not free the resources afterward by setting the object to its default value +(the value the object's state started in). Setting an object `x` back to its +default value is written as `wasMoved(x)`. When not provided the compiler +is using a combination of `=destroy` and `copyMem` instead. This is efficient +hence users rarely need to implement their own `=sink` operator, it is enough to +provide `=destroy` and `=copy`, the compiler will take care of the rest. + +The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=sink`(dest: var T; source: T) + ``` + +The general pattern in `=sink` looks like: + + ```nim + + proc `=sink`(dest: var T; source: T) = + `=destroy`(dest) + wasMoved(dest) + dest.field = source.field + ``` + +**Note**: `=sink` does not need to check for self-assignments. +How self-assignments are handled is explained later in this document. + + +`=copy` hook +------------ + +The ordinary assignment in Nim conceptually copies the values. The `=copy` hook +is called for assignments that couldn't be transformed into `=sink` +operations. + +The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=copy`(dest: var T; source: T) + ``` + +The general pattern in `=copy` looks like: + + ```nim + proc `=copy`(dest: var T; source: T) = + # protect against self-assignments: + if dest.field != source.field: + `=destroy`(dest) + wasMoved(dest) + dest.field = duplicateResource(source.field) + ``` + +The `=copy` proc can be marked with the `{.error.}` pragma. Then any assignment +that otherwise would lead to a copy is prevented at compile-time. This looks like: + + ```nim + proc `=copy`(dest: var T; source: T) {.error.} + ``` + +but a custom error message (e.g., `{.error: "custom error".}`) will not be emitted +by the compiler. Notice that there is no `=` before the `{.error.}` pragma. + + +`=trace` hook +------------- + +A custom **container** type can support Nim's cycle collector `--mm:orc` via +the `=trace` hook. If the container does not implement `=trace`, cyclic data +structures which are constructed with the help of the container might leak +memory or resources, but memory safety is not compromised. + +The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=trace`(dest: var T; env: pointer) + ``` + +`env` is used by ORC to keep track of its internal state, it should be passed around +to calls of the built-in `=trace` operation. + +Usually there will only be a need for a custom `=trace` when a custom `=destroy` that deallocates +manually allocated resources is also used, and then only when there is a chance of cyclic +references from items within the manually allocated resources when it is desired that `--mm:orc` +is able to break and collect these cyclic referenced resources. Currently however, there is a +mutual use problem in that whichever of `=destroy`/`=trace` is used first will automatically +create a version of the other which will then conflict with the creation of the second of the +pair. The workaround for this problem is to forward declare the second of the "hooks" to +prevent the automatic creation. + +The general pattern in using `=destroy` with `=trace` looks like: + + ```nim + type + Test[T] = object + size: Natural + arr: ptr UncheckedArray[T] # raw pointer field + + proc makeTest[T](size: Natural): Test[T] = # custom allocation... + Test[T](size: size, arr: cast[ptr UncheckedArray[T]](alloc0(sizeof(T) * size))) + + + proc `=destroy`[T](dest: Test[T]) = + if dest.arr != nil: + for i in 0 ..< dest.size: dest.arr[i].`=destroy` + dealloc dest.arr + + proc `=trace`[T](dest: var Test[T]; env: pointer) = + if dest.arr != nil: + # trace the `T`'s which may be cyclic + for i in 0 ..< dest.size: `=trace`(dest.arr[i], env) + + # following may be other custom "hooks" as required... + ``` + +**Note**: The `=trace` hooks (which are only used by `--mm:orc`) are currently more experimental and less refined +than the other hooks. + + +`=dup` hook +----------- + +A `=dup` hook duplicates an object. `=dup(x)` can be regarded as an optimization replacing a `wasMoved(dest); =copy(dest, x)` operation. + +The prototype of this hook for a type `T` needs to be: + + ```nim + proc `=dup`(x: T): T + ``` + +The general pattern in implementing `=dup` looks like: + + ```nim + type + Ref[T] = object + data: ptr T + rc: ptr int + + proc `=dup`[T](x: Ref[T]): Ref[T] = + result = x + if x.rc != nil: + inc x.rc[] + ``` + +Move semantics +============== + +A "move" can be regarded as an optimized copy operation. If the source of the +copy operation is not used afterward, the copy can be replaced by a move. This +document uses the notation `lastReadOf(x)` to describe that `x` is not +used afterward. This property is computed by a static control flow analysis +but can also be enforced by using `system.move` explicitly. + +One can query if the analysis is able to perform a move with `system.ensureMove`. +`move` enforces a move operation and calls `=wasMoved` whereas `ensureMove` is +an annotation that implies no runtime operation. An `ensureMove` annotation leads to a static error +if the compiler cannot prove that a move would be safe. + +For example: + + ```nim + proc main(normalParam: string; sinkParam: sink string) = + var x = "abc" + # valid: + let valid = ensureMove x + # invalid: + let invalid = ensureMove normalParam + # valid: + let alsoValid = ensureMove sinkParam + ``` + + +Swap +==== + +The need to check for self-assignments and also the need to destroy previous +objects inside `=copy` and `=sink` is a strong indicator to treat +`system.swap` as a builtin primitive of its own that simply swaps every +field in the involved objects via `copyMem` or a comparable mechanism. +In other words, `swap(a, b)` is **not** implemented +as `let tmp = move(b); b = move(a); a = move(tmp)`. + +This has further consequences: + +* Objects that contain pointers that point to the same object are not supported + by Nim's model. Otherwise swapped objects would end up in an inconsistent state. +* Seqs can use `realloc` in the implementation. + + +Sink parameters +=============== + +To move a variable into a collection usually `sink` parameters are involved. +A location that is passed to a `sink` parameter should not be used afterward. +This is ensured by a static analysis over a control flow graph. If it cannot be +proven to be the last usage of the location, a copy is done instead and this +copy is then passed to the sink parameter. + +A sink parameter +*may* be consumed once in the proc's body but doesn't have to be consumed at all. +The reason for this is that signatures +like `proc put(t: var Table; k: sink Key, v: sink Value)` should be possible +without any further overloads and `put` might not take ownership of `k` if +`k` already exists in the table. Sink parameters enable an affine type system, +not a linear type system. + +The employed static analysis is limited and only concerned with local variables; +however, object and tuple fields are treated as separate entities: + + ```nim + proc consume(x: sink Obj) = discard "no implementation" + + proc main = + let tup = (Obj(), Obj()) + consume tup[0] + # ok, only tup[0] was consumed, tup[1] is still alive: + echo tup[1] + ``` + +Sometimes it is required to explicitly `move` a value into its final position: + + ```nim + proc main = + var dest, src: array[10, string] + # ... + for i in 0..high(dest): dest[i] = move(src[i]) + ``` + +An implementation is allowed, but not required to implement even more move +optimizations (and the current implementation does not). + + +Sink parameter inference +======================== + +The current implementation can do a limited form of sink parameter +inference. But it has to be enabled via `--sinkInference:on`:option:, either +on the command line or via a `push` pragma. + +To enable it for a section of code, one can +use `{.push sinkInference: on.}` ... `{.pop.}`. + +The `.nosinks`:idx: pragma can be used to disable this inference +for a single routine: + + ```nim + proc addX(x: T; child: T) {.nosinks.} = + x.s.add child + ``` + +The details of the inference algorithm are currently undocumented. + + +Rewrite rules +============= + +**Note**: There are two different allowed implementation strategies: + +1. The produced `finally` section can be a single section that is wrapped + around the complete routine body. +2. The produced `finally` section is wrapped around the enclosing scope. + +The current implementation follows strategy (2). This means that resources are +destroyed at the scope exit. + + + var x: T; stmts + --------------- (destroy-var) + var x: T; try stmts + finally: `=destroy`(x) + + + g(f(...)) + ------------------------ (nested-function-call) + g(let tmp; + bitwiseCopy tmp, f(...); + tmp) + finally: `=destroy`(tmp) + + + x = f(...) + ------------------------ (function-sink) + `=sink`(x, f(...)) + + + x = lastReadOf z + ------------------ (move-optimization) + `=sink`(x, z) + `=wasMoved`(z) + + + v = v + ------------------ (self-assignment-removal) + discard "nop" + + + x = y + ------------------ (copy) + `=copy`(x, y) + + + f_sink(g()) + ----------------------- (call-to-sink) + f_sink(g()) + + + f_sink(notLastReadOf y) + -------------------------- (copy-to-sink) + (let tmp = `=dup`(y); + f_sink(tmp)) + + + f_sink(lastReadOf y) + ----------------------- (move-to-sink) + f_sink(y) + `=wasMoved`(y) + + +Object and array construction +============================= + +Object and array construction is treated as a function call where the +function has `sink` parameters. + + +Destructor removal +================== + +`=wasMoved(x)` followed by a `=destroy(x)` operation cancel each other +out. An implementation is encouraged to exploit this in order to improve +efficiency and code sizes. The current implementation does perform this +optimization. + + +Self assignments +================ + +`=sink` in combination with `=wasMoved` can handle self-assignments but +it's subtle. + +The simple case of `x = x` cannot be turned +into `=sink(x, x); =wasMoved(x)` because that would lose `x`'s value. +The solution is that simple self-assignments that consist of + +- Symbols: `x = x` +- Field access: `x.f = x.f` +- Array, sequence or string access with indices known at compile-time: `x[0] = x[0]` + +are transformed into an empty statement that does nothing. +The compiler is free to optimize further cases. + +The complex case looks like a variant of `x = f(x)`, we consider +`x = select(rand() < 0.5, x, y)` here: + + + ```nim + proc select(cond: bool; a, b: sink string): string = + if cond: + result = a # moves a into result + else: + result = b # moves b into result + + proc main = + var x = "abc" + var y = "xyz" + # possible self-assignment: + x = select(true, x, y) + ``` + +Is transformed into: + + ```nim + proc select(cond: bool; a, b: sink string): string = + try: + if cond: + `=sink`(result, a) + `=wasMoved`(a) + else: + `=sink`(result, b) + `=wasMoved`(b) + finally: + `=destroy`(b) + `=destroy`(a) + + proc main = + var + x: string + y: string + try: + `=sink`(x, "abc") + `=sink`(y, "xyz") + `=sink`(x, select(true, + let blitTmp = x + `=wasMoved`(x) + blitTmp, + let blitTmp = y + `=wasMoved`(y) + blitTmp)) + echo [x] + finally: + `=destroy`(y) + `=destroy`(x) + ``` + +As can be manually verified, this transformation is correct for +self-assignments. + + +Lent type +========= + +`proc p(x: sink T)` means that the proc `p` takes ownership of `x`. +To eliminate even more creation/copy <-> destruction pairs, a proc's return +type can be annotated as `lent T`. This is useful for "getter" accessors +that seek to allow an immutable view into a container. + +The `sink` and `lent` annotations allow us to remove most (if not all) +superfluous copies and destructions. + +`lent T` is like `var T` a hidden pointer. It is proven by the compiler +that the pointer does not outlive its origin. No destructor call is injected +for expressions of type `lent T` or of type `var T`. + + + ```nim test + type + Tree = object + kids: seq[Tree] + + proc construct(kids: sink seq[Tree]): Tree = + result = Tree(kids: kids) + # converted into: + `=sink`(result.kids, kids); `=wasMoved`(kids) + `=destroy`(kids) + + proc `[]`*(x: Tree; i: int): lent Tree = + result = x.kids[i] + # borrows from 'x', this is transformed into: + # result = addr x.kids[i] + # This means 'lent' is like 'var T' a hidden pointer. + # Unlike 'var' this hidden pointer cannot be used to mutate the object. + + iterator children*(t: Tree): lent Tree = + for x in t.kids: yield x + + proc main = + # everything turned into moves: + let t = construct(@[construct(@[]), construct(@[])]) + echo t[0] # accessor does not copy the element! + ``` + + +The cursor pragma +================= + +Under the `--mm:arc|orc`:option: modes Nim's `ref` type is implemented +via the same runtime "hooks" and thus via reference counting. +This means that cyclic structures cannot be freed +immediately (`--mm:orc`:option: ships with a cycle collector). +With the `cursor` pragma one can break up cycles declaratively: + + ```nim + type + Node = ref object + left: Node # owning ref + right {.cursor.}: Node # non-owning ref + ``` + +But please notice that this is not C++'s weak_ptr, it means the right field is not +involved in the reference counting, it is a raw pointer without runtime checks. + +Automatic reference counting also has the disadvantage that it introduces overhead +when iterating over linked structures. The `cursor` pragma can also be used +to avoid this overhead: + + ```nim + var it {.cursor.} = listRoot + while it != nil: + use(it) + it = it.next + ``` + +In fact, `cursor` more generally prevents object construction/destruction pairs +and so can also be useful in other contexts. The alternative solution would be to +use raw pointers (`ptr`) instead which is more cumbersome and also more dangerous +for Nim's evolution: Later on, the compiler can try to prove `cursor` pragmas +to be safe, but for `ptr` the compiler has to remain silent about possible +problems. + + +Cursor inference / copy elision +=============================== + +The current implementation also performs `cursor` inference. Cursor inference is +a form of copy elision. + +To see how and when we can do that, think about this question: In `dest = src` when +do we really have to *materialize* the full copy? - Only if `dest` or `src` are mutated +afterward. If `dest` is a local variable that is simple to analyze. And if `src` is a +location derived from a formal parameter, we also know it is not mutated! In other +words, we do a compile-time copy-on-write analysis. + +This means that "borrowed" views can be written naturally and without explicit pointer +indirections: + + ```nim + proc main(tab: Table[string, string]) = + let v = tab["key"] # inferred as cursor because 'tab' is not mutated. + # no copy into 'v', no destruction of 'v'. + use(v) + useItAgain(v) + ``` + + +Hook lifting +============ + +The hooks of a tuple type `(A, B, ...)` are generated by lifting the +hooks of the involved types `A`, `B`, ... to the tuple type. In +other words, a copy `x = y` is implemented +as `x[0] = y[0]; x[1] = y[1]; ...`, likewise for `=sink` and `=destroy`. + +Other value-based compound types like `object` and `array` are handled +correspondingly. For `object` however, the compiler-generated hooks +can be overridden. This can also be important to use an alternative traversal +of the involved data structure that is more efficient or in order to avoid +deep recursions. + + + +Hook generation +=============== + +The ability to override a hook leads to a phase ordering problem: + + ```nim + type + Foo[T] = object + + proc main = + var f: Foo[int] + # error: destructor for 'f' called here before + # it was seen in this module. + + proc `=destroy`[T](f: Foo[T]) = + discard + ``` + +The solution is to define ``proc `=destroy`[T](f: Foo[T])`` before +it is used. The compiler generates implicit +hooks for all types in *strategic places* so that an explicitly provided +hook that comes too "late" can be detected reliably. These *strategic places* +have been derived from the rewrite rules and are as follows: + +- In the construct `let/var x = ...` (var/let binding) + hooks are generated for `typeof(x)`. +- In `x = ...` (assignment) hooks are generated for `typeof(x)`. +- In `f(...)` (function call) hooks are generated for `typeof(f(...))`. +- For every sink parameter `x: sink T` the hooks are generated + for `typeof(x)`. + + +nodestroy pragma +================ + +The experimental `nodestroy`:idx: pragma inhibits hook injections. This can be +used to specialize the object traversal in order to avoid deep recursions: + + + ```nim test + type Node = ref object + x, y: int32 + left, right: Node + + type Tree = object + root: Node + + proc `=destroy`(t: Tree) {.nodestroy.} = + # use an explicit stack so that we do not get stack overflows: + var s: seq[Node] = @[t.root] + while s.len > 0: + let x = s.pop + if x.left != nil: s.add(x.left) + if x.right != nil: s.add(x.right) + # free the memory explicitly: + `=dispose`(x) + # notice how even the destructor for 's' is not called implicitly + # anymore thanks to .nodestroy, so we have to call it on our own: + `=destroy`(s) + ``` + + +As can be seen from the example, this solution is hardly sufficient and +should eventually be replaced by a better solution. + + +Copy on write +============= + +String literals are implemented as "copy on write". +When assigning a string literal to a variable, a copy of the literal won't be created. +Instead the variable simply points to the literal. +The literal is shared between different variables which are pointing to it. +The copy operation is deferred until the first write. + +For example: + + ```nim + var x = "abc" # no copy + var y = x # no copy + y[0] = 'h' # copy + ``` + +The abstraction fails for `addr x` because whether the address is going to be used for mutations is unknown. +`prepareMutation` needs to be called before the "address of" operation. For example: + + ```nim + var x = "abc" + var y = x + + prepareMutation(y) + moveMem(addr y[0], addr x[0], 3) + assert y == "abc" + ``` diff --git a/doc/destructors.rst b/doc/destructors.rst deleted file mode 100644 index 4285bad8b..000000000 --- a/doc/destructors.rst +++ /dev/null @@ -1,632 +0,0 @@ -================================== -Nim Destructors and Move Semantics -================================== - -:Authors: Andreas Rumpf -:Version: |nimversion| - -.. contents:: - - -About this document -=================== - -This document describes the upcoming Nim runtime which does -not use classical GC algorithms anymore but is based on destructors and -move semantics. The new runtime's advantages are that Nim programs become -oblivious to the involved heap sizes and programs are easier to write to make -effective use of multi-core machines. As a nice bonus, files and sockets and -the like will not require manual ``close`` calls anymore. - -This document aims to be a precise specification about how -move semantics and destructors work in Nim. - - -Motivating example -================== - -With the language mechanisms described here a custom seq could be -written as: - -.. code-block:: nim - - type - myseq*[T] = object - len, cap: int - data: ptr UncheckedArray[T] - - proc `=destroy`*[T](x: var myseq[T]) = - if x.data != nil: - for i in 0..<x.len: `=destroy`(x[i]) - dealloc(x.data) - x.data = nil - - proc `=`*[T](a: var myseq[T]; b: myseq[T]) = - # do nothing for self-assignments: - if a.data == b.data: return - `=destroy`(a) - a.len = b.len - a.cap = b.cap - if b.data != nil: - a.data = cast[typeof(a.data)](alloc(a.cap * sizeof(T))) - for i in 0..<a.len: - a.data[i] = b.data[i] - - proc `=sink`*[T](a: var myseq[T]; b: myseq[T]) = - # move assignment, optional. - # Compiler is using `=destroy` and `copyMem` when not provided - `=destroy`(a) - a.len = b.len - a.cap = b.cap - a.data = b.data - - proc add*[T](x: var myseq[T]; y: sink T) = - if x.len >= x.cap: resize(x) - x.data[x.len] = y - inc x.len - - proc `[]`*[T](x: myseq[T]; i: Natural): lent T = - assert i < x.len - x.data[i] - - proc `[]=`*[T](x: var myseq[T]; i: Natural; y: sink T) = - assert i < x.len - x.data[i] = y - - proc createSeq*[T](elems: varargs[T]): myseq[T] = - result.cap = elems.len - result.len = elems.len - result.data = cast[typeof(result.data)](alloc(result.cap * sizeof(T))) - for i in 0..<result.len: result.data[i] = elems[i] - - proc len*[T](x: myseq[T]): int {.inline.} = x.len - - - -Lifetime-tracking hooks -======================= - -The memory management for Nim's standard ``string`` and ``seq`` types as -well as other standard collections is performed via so called -"Lifetime-tracking hooks" or "type-bound operators". There are 3 different -hooks for each (generic or concrete) object type ``T`` (``T`` can also be a -``distinct`` type) that are called implicitly by the compiler. - -(Note: The word "hook" here does not imply any kind of dynamic binding -or runtime indirections, the implicit calls are statically bound and -potentially inlined.) - - -`=destroy` hook ---------------- - -A `=destroy` hook frees the object's associated memory and releases -other associated resources. Variables are destroyed via this hook when -they go out of scope or when the routine they were declared in is about -to return. - -The prototype of this hook for a type ``T`` needs to be: - -.. code-block:: nim - - proc `=destroy`(x: var T) - - -The general pattern in ``=destroy`` looks like: - -.. code-block:: nim - - proc `=destroy`(x: var T) = - # first check if 'x' was moved to somewhere else: - if x.field != nil: - freeResource(x.field) - x.field = nil - - - -`=sink` hook ------------- - -A `=sink` hook moves an object around, the resources are stolen from the source -and passed to the destination. It is ensured that source's destructor does -not free the resources afterwards by setting the object to its default value -(the value the object's state started in). Setting an object ``x`` back to its -default value is written as ``wasMoved(x)``. When not provided the compiler -is using a combination of `=destroy` and `copyMem` instead. This is efficient -hence users rarely need to implement their own `=sink` operator, it is enough to -provide `=destroy` and `=`, compiler will take care about the rest. - -The prototype of this hook for a type ``T`` needs to be: - -.. code-block:: nim - - proc `=sink`(dest: var T; source: T) - - -The general pattern in ``=sink`` looks like: - -.. code-block:: nim - - proc `=sink`(dest: var T; source: T) = - `=destroy`(dest) - dest.field = source.field - - -**Note**: ``=sink`` does not need to check for self-assignments. -How self-assignments are handled is explained later in this document. - - -`=` (copy) hook ---------------- - -The ordinary assignment in Nim conceptually copies the values. The ``=`` hook -is called for assignments that couldn't be transformed into ``=sink`` -operations. - -The prototype of this hook for a type ``T`` needs to be: - -.. code-block:: nim - - proc `=`(dest: var T; source: T) - - -The general pattern in ``=`` looks like: - -.. code-block:: nim - - proc `=`(dest: var T; source: T) = - # protect against self-assignments: - if dest.field != source.field: - `=destroy`(dest) - dest.field = duplicateResource(source.field) - - -The ``=`` proc can be marked with the ``{.error.}`` pragma. Then any assignment -that otherwise would lead to a copy is prevented at compile-time. - - -Move semantics -============== - -A "move" can be regarded as an optimized copy operation. If the source of the -copy operation is not used afterwards, the copy can be replaced by a move. This -document uses the notation ``lastReadOf(x)`` to describe that ``x`` is not -used afterwards. This property is computed by a static control flow analysis -but can also be enforced by using ``system.move`` explicitly. - - -Swap -==== - -The need to check for self-assignments and also the need to destroy previous -objects inside ``=`` and ``=sink`` is a strong indicator to treat -``system.swap`` as a builtin primitive of its own that simply swaps every -field in the involved objects via ``copyMem`` or a comparable mechanism. -In other words, ``swap(a, b)`` is **not** implemented -as ``let tmp = move(b); b = move(a); a = move(tmp)``. - -This has further consequences: - -* Objects that contain pointers that point to the same object are not supported - by Nim's model. Otherwise swapped objects would end up in an inconsistent state. -* Seqs can use ``realloc`` in the implementation. - - -Sink parameters -=============== - -To move a variable into a collection usually ``sink`` parameters are involved. -A location that is passed to a ``sink`` parameter should not be used afterwards. -This is ensured by a static analysis over a control flow graph. If it cannot be -proven to be the last usage of the location, a copy is done instead and this -copy is then passed to the sink parameter. - -A sink parameter -*may* be consumed once in the proc's body but doesn't have to be consumed at all. -The reason for this is that signatures -like ``proc put(t: var Table; k: sink Key, v: sink Value)`` should be possible -without any further overloads and ``put`` might not take ownership of ``k`` if -``k`` already exists in the table. Sink parameters enable an affine type system, -not a linear type system. - -The employed static analysis is limited and only concerned with local variables; -however object and tuple fields are treated as separate entities: - -.. code-block:: nim - - proc consume(x: sink Obj) = discard "no implementation" - - proc main = - let tup = (Obj(), Obj()) - consume tup[0] - # ok, only tup[0] was consumed, tup[1] is still alive: - echo tup[1] - - -Sometimes it is required to explicitly ``move`` a value into its final position: - -.. code-block:: nim - - proc main = - var dest, src: array[10, string] - # ... - for i in 0..high(dest): dest[i] = move(src[i]) - -An implementation is allowed, but not required to implement even more move -optimizations (and the current implementation does not). - - -Sink parameter inference -======================== - -The current implementation does a limited form of sink parameter -inference. The `.nosinks`:idx: pragma can be used to disable this inference -for a single routine: - -.. code-block:: nim - - proc addX(x: T; child: T) {.nosinks.} = - x.s.add child - -To disable it for a section of code, one can -use `{.push sinkInference: off.}`...`{.pop.}`. - -The details of the inference algorithm are currently undocumented. - - -Rewrite rules -============= - -**Note**: There are two different allowed implementation strategies: - -1. The produced ``finally`` section can be a single section that is wrapped - around the complete routine body. -2. The produced ``finally`` section is wrapped around the enclosing scope. - -The current implementation follows strategy (2). This means that resources are -destroyed at the scope exit. - -:: - - var x: T; stmts - --------------- (destroy-var) - var x: T; try stmts - finally: `=destroy`(x) - - - g(f(...)) - ------------------------ (nested-function-call) - g(let tmp; - bitwiseCopy tmp, f(...); - tmp) - finally: `=destroy`(tmp) - - - x = f(...) - ------------------------ (function-sink) - `=sink`(x, f(...)) - - - x = lastReadOf z - ------------------ (move-optimization) - `=sink`(x, z) - wasMoved(z) - - - v = v - ------------------ (self-assignment-removal) - discard "nop" - - - x = y - ------------------ (copy) - `=`(x, y) - - - f_sink(g()) - ----------------------- (call-to-sink) - f_sink(g()) - - - f_sink(notLastReadOf y) - -------------------------- (copy-to-sink) - (let tmp; `=`(tmp, y); - f_sink(tmp)) - - - f_sink(lastReadOf y) - ----------------------- (move-to-sink) - f_sink(y) - wasMoved(y) - - -Object and array construction -============================= - -Object and array construction is treated as a function call where the -function has ``sink`` parameters. - - -Destructor removal -================== - -``wasMoved(x);`` followed by a `=destroy(x)` operation cancel each other -out. An implementation is encouraged to exploit this in order to improve -efficiency and code sizes. - - -Self assignments -================ - -``=sink`` in combination with ``wasMoved`` can handle self-assignments but -it's subtle. - -The simple case of ``x = x`` cannot be turned -into ``=sink(x, x); wasMoved(x)`` because that would lose ``x``'s value. -The solution is that simple self-assignments are simply transformed into -an empty statement that does nothing. - -The complex case looks like a variant of ``x = f(x)``, we consider -``x = select(rand() < 0.5, x, y)`` here: - - -.. code-block:: nim - - proc select(cond: bool; a, b: sink string): string = - if cond: - result = a # moves a into result - else: - result = b # moves b into result - - proc main = - var x = "abc" - var y = "xyz" - # possible self-assignment: - x = select(true, x, y) - - -Is transformed into: - - -.. code-block:: nim - - proc select(cond: bool; a, b: sink string): string = - try: - if cond: - `=sink`(result, a) - wasMoved(a) - else: - `=sink`(result, b) - wasMoved(b) - finally: - `=destroy`(b) - `=destroy`(a) - - proc main = - var - x: string - y: string - try: - `=sink`(x, "abc") - `=sink`(y, "xyz") - `=sink`(x, select(true, - let blitTmp = x - wasMoved(x) - blitTmp, - let blitTmp = y - wasMoved(y) - blitTmp)) - echo [x] - finally: - `=destroy`(y) - `=destroy`(x) - -As can be manually verified, this transformation is correct for -self-assignments. - - -Lent type -========= - -``proc p(x: sink T)`` means that the proc ``p`` takes ownership of ``x``. -To eliminate even more creation/copy <-> destruction pairs, a proc's return -type can be annotated as ``lent T``. This is useful for "getter" accessors -that seek to allow an immutable view into a container. - -The ``sink`` and ``lent`` annotations allow us to remove most (if not all) -superfluous copies and destructions. - -``lent T`` is like ``var T`` a hidden pointer. It is proven by the compiler -that the pointer does not outlive its origin. No destructor call is injected -for expressions of type ``lent T`` or of type ``var T``. - - -.. code-block:: nim - - type - Tree = object - kids: seq[Tree] - - proc construct(kids: sink seq[Tree]): Tree = - result = Tree(kids: kids) - # converted into: - `=sink`(result.kids, kids); wasMoved(kids) - `=destroy`(kids) - - proc `[]`*(x: Tree; i: int): lent Tree = - result = x.kids[i] - # borrows from 'x', this is transformed into: - result = addr x.kids[i] - # This means 'lent' is like 'var T' a hidden pointer. - # Unlike 'var' this hidden pointer cannot be used to mutate the object. - - iterator children*(t: Tree): lent Tree = - for x in t.kids: yield x - - proc main = - # everything turned into moves: - let t = construct(@[construct(@[]), construct(@[])]) - echo t[0] # accessor does not copy the element! - - -The .cursor annotation -====================== - -Under the ``--gc:arc|orc`` modes Nim's `ref` type is implemented via the same runtime -"hooks" and thus via reference counting. This means that cyclic structures cannot be freed -immediately (``--gc:orc`` ships with a cycle collector). With the ``.cursor`` annotation -one can break up cycles declaratively: - -.. code-block:: nim - - type - Node = ref object - left: Node # owning ref - right {.cursor.}: Node # non-owning ref - -But please notice that this is not C++'s weak_ptr, it means the right field is not -involved in the reference counting, it is a raw pointer without runtime checks. - -Automatic reference counting also has the disadvantage that it introduces overhead -when iterating over linked structures. The ``.cursor`` annotation can also be used -to avoid this overhead: - -.. code-block:: nim - - var it {.cursor.} = listRoot - while it != nil: - use(it) - it = it.next - - -In fact, ``.cursor`` more generally prevents object construction/destruction pairs -and so can also be useful in other contexts. The alternative solution would be to -use raw pointers (``ptr``) instead which is more cumbersome and also more dangerous -for Nim's evolution: Later on the compiler can try to prove ``.cursor`` annotations -to be safe, but for ``ptr`` the compiler has to remain silent about possible -problems. - - -Owned refs -========== - -**Note**: The ``owned`` type constructor is only available with -the ``--newruntime`` compiler switch and is experimental. - - -Let ``W`` be an ``owned ref`` type. Conceptually its hooks look like: - -.. code-block:: nim - - proc `=destroy`(x: var W) = - if x != nil: - assert x.refcount == 0, "dangling unowned pointers exist!" - `=destroy`(x[]) - x = nil - - proc `=`(x: var W; y: W) {.error: "owned refs can only be moved".} - - proc `=sink`(x: var W; y: W) = - `=destroy`(x) - bitwiseCopy x, y # raw pointer copy - - -Let ``U`` be an unowned ``ref`` type. Conceptually its hooks look like: - -.. code-block:: nim - - proc `=destroy`(x: var U) = - if x != nil: - dec x.refcount - - proc `=`(x: var U; y: U) = - # Note: No need to check for self-assignments here. - if y != nil: inc y.refcount - if x != nil: dec x.refcount - bitwiseCopy x, y # raw pointer copy - - proc `=sink`(x: var U, y: U) {.error.} - # Note: Moves are not available. - - -Hook lifting -============ - -The hooks of a tuple type ``(A, B, ...)`` are generated by lifting the -hooks of the involved types ``A``, ``B``, ... to the tuple type. In -other words, a copy ``x = y`` is implemented -as ``x[0] = y[0]; x[1] = y[1]; ...``, likewise for ``=sink`` and ``=destroy``. - -Other value-based compound types like ``object`` and ``array`` are handled -correspondingly. For ``object`` however, the compiler generated hooks -can be overridden. This can also be important to use an alternative traversal -of the involved datastructure that is more efficient or in order to avoid -deep recursions. - - - -Hook generation -=============== - -The ability to override a hook leads to a phase ordering problem: - -.. code-block:: nim - - type - Foo[T] = object - - proc main = - var f: Foo[int] - # error: destructor for 'f' called here before - # it was seen in this module. - - proc `=destroy`[T](f: var Foo[T]) = - discard - - -The solution is to define ``proc `=destroy`[T](f: var Foo[T])`` before -it is used. The compiler generates implicit -hooks for all types in *strategic places* so that an explicitly provided -hook that comes too "late" can be detected reliably. These *strategic places* -have been derived from the rewrite rules and are as follows: - -- In the construct ``let/var x = ...`` (var/let binding) - hooks are generated for ``typeof(x)``. -- In ``x = ...`` (assignment) hooks are generated for ``typeof(x)``. -- In ``f(...)`` (function call) hooks are generated for ``typeof(f(...))``. -- For every sink parameter ``x: sink T`` the hooks are generated - for ``typeof(x)``. - - -nodestroy pragma -================ - -The experimental `nodestroy`:idx: pragma inhibits hook injections. This can be -used to specialize the object traversal in order to avoid deep recursions: - - -.. code-block:: nim - - type Node = ref object - x, y: int32 - left, right: Node - - type Tree = object - root: Node - - proc `=destroy`(t: var Tree) {.nodestroy.} = - # use an explicit stack so that we do not get stack overflows: - var s: seq[Node] = @[t.root] - while s.len > 0: - let x = s.pop - if x.left != nil: s.add(x.left) - if x.right != nil: s.add(x.right) - # free the memory explicit: - dispose(x) - # notice how even the destructor for 's' is not called implicitly - # anymore thanks to .nodestroy, so we have to call it on our own: - `=destroy`(s) - - -As can be seen from the example, this solution is hardly sufficient and -should eventually be replaced by a better solution. diff --git a/doc/docgen.md b/doc/docgen.md new file mode 100644 index 000000000..3cc75fc18 --- /dev/null +++ b/doc/docgen.md @@ -0,0 +1,891 @@ +=================================== + Nim DocGen Tools Guide +=================================== + +:Author: Erik O'Leary +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +.. importdoc:: markdown_rst.md, compiler/docgen.nim + +Introduction +============ + +This document describes the `documentation generation tools`:idx: built into +the [Nim compiler](nimc.html), which can generate HTML, Latex and JSON output +from input ``.nim`` files and projects. +The output documentation will include the module +dependencies (`import`), any top-level documentation comments (`##`), and +exported symbols (`*`), including procedures, types, and variables. + +=================== ============== +command output format +=================== ============== +`nim doc`:cmd: ``.html`` HTML +`nim doc2tex`:cmd: ``.tex`` LaTeX +`nim jsondoc`:cmd: ``.json`` JSON +=================== ============== + +Nim can generate HTML and LaTeX from input Markdown and +RST (reStructuredText) files as well, which is intended for writing +standalone documents like user's guides and technical specifications. +See [Nim-flavored Markdown and reStructuredText] document for the description +of this feature and particularly section [Command line usage] for the full +list of supported commands. + +Quick start +----------- + +Generate HTML documentation for a file: + + ```cmd + nim doc <filename>.nim + ``` + +Generate HTML documentation for a whole project: + + ```cmd + # delete any htmldocs/*.idx file before starting + nim doc --project --index:on --git.url:<url> --git.commit:<tag> --outdir:htmldocs <main_filename>.nim + # this will generate html files, a theindex.html index, css and js under `htmldocs` + # See also `--docroot` to specify a relative root. + # to get search (dochacks.js) to work locally, you need a server otherwise + # CORS will prevent opening file:// urls; this works: + python3 -m http.server 7029 --directory htmldocs + # When --outdir is omitted it defaults to $projectPath/htmldocs, + # or `$nimcache/htmldocs` with `--usenimcache` which avoids clobbering your sources; + # and likewise without `--project`. + # Adding `-r` will open in a browser directly. + # Use `--showNonExports` to show non-exported fields of an exported type. + ``` + +Documentation Comments +---------------------- + +Any comments which are preceded by a double-hash (`##`), are interpreted as +documentation. Comments are parsed as RST (see [reference]( +http://docutils.sourceforge.net/docs/user/rst/quickref.html)), providing +Nim module authors the ability to easily generate richly formatted +documentation with only their well-documented code! +Basic Markdown syntax is also supported inside the doc comments. + +Example: + + ```nim + type Person* = object + ## This type contains a description of a person + name: string + age: int + ``` + +Outputs: + + Person* = object + name: string + age: int + + This type contains a description of a person + +Field documentation comments can be added to fields like so: + + ```nim + var numValues: int ## \ + ## `numValues` stores the number of values + ``` + +Note that without the `*` following the name of the type, the documentation for +this type would not be generated. Documentation will only be generated for +*exported* types/procedures/etc. + +It's recommended to always add exactly **one** space after `##` for readability +of comments — this extra space will be cropped from the parsed comments and +won't influence RST formatting. + +.. note:: Generally, this baseline indentation level inside a documentation + comment may not be 1: it can be any since it is determined by the offset + of the first non-whitespace character in the comment. + After that indentation **must** be consistent on the following lines of + the same comment. + If you still need to add an additional indentation at the very beginning + (for RST block quote syntax) use backslash \\ before it: + + ```nim + ## \ + ## + ## Block quote at the first line. + ## + ## Paragraph. + ``` + +Structuring output directories +------------------------------ + +Basic directory for output is set by `--outdir:OUTDIR`:option: switch, +by default `OUTDIR` is ``htmldocs`` sub-directory in the directory of +the processed file. + +There are 2 basic options as to how generated HTML output files are stored: + +1) complex hierarchy when docgen-compiling with `--project`:option:, + which follows directory structure of the project itself. + So `nim doc`:cmd: replicates project's directory structure + inside `--outdir:OUTDIR`:option: directory. + `--project`:option: is well suited for projects that have 1 main module. + File name clashes are impossible in this case. + +2) flattened structure, where user-provided script goes through all + needed input files and calls commands like `nim doc`:cmd: + with `--outdir:OUTDIR`:option: switch, thus putting all HTML (and + ``.idx``) files into 1 directory. + + .. Important:: Make sure that you don't have files with same base name + like ``x.nim`` and ``x.md`` in the same package, otherwise you'll + have name conflict for ``x.html``. + + .. Tip:: To structure your output directories and avoid file name + clashes you can split your project into + different *packages* -- parts of your repository that are + docgen-compiled with different `--outdir:OUTDIR`:option: options. + + An example of such strategy is Nim repository itself which has: + + * its stdlib ``.nim`` files from different directories and ``.md`` + documentation from ``doc/`` directory are all docgen-compiled + into `--outdir:web/upload/<version>/`:option: directory + * its ``.nim`` files from ``compiler/`` directory are docgen-compiled + into `--outdir:web/upload/<version>/compiler/`:option: directory. + Interestingly, it's compiled with complex hierarchy using + `--project`:option: switch. + + Contents of ``web/upload/<version>`` are then deployed into Nim's + Web server. + + This output directory structure allows to work correctly with files like + ``compiler/docgen.nim`` (implementation) and ``doc/docgen.md`` (user + documentation) in 1 repository. + + +Index files +----------- + +Index (``.idx``) files are used for 2 different purposes: + +1. easy cross-referencing between different ``.nim`` and/or ``.md`` / ``.rst`` + files described in [Nim external referencing] +2. creating a whole-project index for searching of symbols and keywords, + see [Buildindex command]. + + +Document Types +============== + +Example of Nim file input +------------------------- + +The following examples will generate documentation for this sample +*Nim* module, aptly named ``doc/docgen_sample.nim``: + + ```nim file=docgen_sample.nim + ``` + +All the below commands save their output to ``htmldocs`` directory relative to +the directory of file; +hence the output for this sample will be in ``doc/htmldocs``. + +HTML +---- + +The generation of HTML documents is done via the `doc`:option: command. This command +takes either a single ``.nim`` file, outputting a single ``.html`` file with the same +base filename, or multiple ``.nim`` files, outputting multiple ``.html`` files and, +optionally, an index file. + +The `doc`:option: command: + + ```cmd + nim doc docgen_sample.nim + ``` + +Partial Output: + + ... + proc helloWorld(times: int) {.raises: [], tags: [].} + ... + +The full output can be seen here: [docgen_sample.html](docgen_sample.html). +It runs after semantic checking and includes pragmas attached implicitly by the +compiler. + +LaTeX +----- + +LaTeX files are intended to be converted to PDF, especially for offline +reading or making hard copies. (LaTeX output is oftentimes better than +HTML -> PDF conversion). + +The `doc2tex`:option: command: + + ```cmd + nim doc2tex docgen_sample.nim + cd htmldocs + xelatex docgen_sample.tex + xelatex docgen_sample.tex + # It is usually necessary to run `xelatex` 2 times (or even 3 times for + # large documents) to get all labels generated. + # That depends on this warning in the end of `xelatex` output: + # LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right. + ``` + +The output is ``docgen_sample.pdf``. + +JSON +---- + +The generation of JSON documents is done via the `jsondoc`:option: command. +This command takes in a ``.nim`` file and outputs a ``.json`` file with +the same base filename. +Note that this tool is built off of the `doc`:option: command +(previously `doc2`:option:), and contains the same information. + +The `jsondoc`:option: command: + + ```cmd + nim jsondoc docgen_sample.nim + ``` + +Output: + + { + "orig": "docgen_sample.nim", + "nimble": "", + "moduleDescription": "This module is a sample", + "entries": [ + { + "name": "helloWorld", + "type": "skProc", + "line": 5, + "col": 0, + "description": "Takes an integer and outputs as many "hello world!"s", + "code": "proc helloWorld(times: int) {.raises: [], tags: [].}" + } + ] + } + +Similarly to the old `doc`:option: command, the old `jsondoc`:option: command has been +renamed to `jsondoc0`:option:. + +The `jsondoc0`:option: command: + + ```cmd + nim jsondoc0 docgen_sample.nim + ``` + +Output: + + [ + { + "comment": "This module is a sample." + }, + { + "name": "helloWorld", + "type": "skProc", + "description": "Takes an integer and outputs as many "hello world!"s", + "code": "proc helloWorld*(times: int)" + } + ] + +Note that the `jsondoc`:option: command outputs its JSON without pretty-printing it, +while `jsondoc0`:option: outputs pretty-printed JSON. + + +Simple documentation links +========================== + +It's possible to use normal Markdown/RST syntax to *manually* +reference Nim symbols using HTML anchors, however Nim has an *automatic* +facility that makes referencing inside ``.nim`` and ``.md/.rst`` files and +between them easy and seamless. +The point is that such links will be resolved automatically +by `nim doc`:cmd: (or `md2html`:option:, or `jsondoc`:option:, +or `doc2tex`:option:, ...). And, unlike manual links, such automatic +links **check** that their target exists -- a warning is emitted for +any broken link, so you avoid broken links in your project. + +Nim treats both ``.md/.rst`` files and ``.nim`` modules (their doc comment +part) as *documents* uniformly. +Hence all directions of referencing are equally possible having the same syntax: + +1. ``.md/rst`` -> itself (internal). See [Markup local referencing]. +2. ``.md/rst`` -> external ``.md/rst``. See [Markup external referencing]. + To summarize, referencing in `.md`/`.rst` files was already described in + [Nim-flavored Markdown and reStructuredText] + (particularly it described usage of index files for referencing), + while in this document we focus on Nim-specific details. +3. ``.md/rst`` -> external ``.nim``. See [Nim external referencing]. +4. ``.nim`` -> itself (internal). See [Nim local referencing]. +5. ``.nim`` -> external ``.md/rst``. See [Markup external referencing]. +6. ``.nim`` -> external ``.nim``. See [Nim external referencing]. + +To put it shortly, local referencing always works out of the box, +external referencing requires to use ``.. importdoc:: <file>`` +directive to import `file` and to ensure that the corresponding +``.idx`` file was generated. + +Syntax for referencing is basically the same as for normal markup. +Recall from [Referencing] that our parser supports two equivalent syntaxes +for referencing, Markdown and RST one. +So to reference ``proc f`` one should use something like that, +depending on markup type: + + Markdown RST + + Ref. [proc f] or [f] Ref. `proc f`_ or just f_ for a one-word case + +Nim local referencing +--------------------- + +You can reference Nim identifiers from Nim documentation comments +inside their ``.nim`` file (or inside a ``.rst`` file included from +a ``.nim``). +This pertains to any exported symbol like `proc`, `const`, `iterator`, etc. +Link text is either one word or a group of words enclosed by delimiters +(brackets ``[...]`` for Markdown or backticks `\`...\`_` for RST). +Link text will be displayed *as is* while *link target* will be set to +the anchor [^1] of Nim symbol that corresponds to link text. + +[^1] anchors' format is described in [HTML anchor generation] section below. + +If you have a constant: + + ```Nim + const pi* = 3.14 + ``` + +then it should be referenced in one of the 2 forms: + +A. non-qualified (no symbol kind specification): + + pi_ + +B. qualified (with symbol kind specification): + + `const pi`_ + +For routine kinds there are more options. Consider this definition: + + ```Nim + proc foo*(a: int, b: float): string + ``` + +Generally following syntax is allowed for referencing `foo`: + +* short (without parameters): + + A. non-qualified: + + foo_ + + B. qualified: + + `proc foo`_ + +* longer variants (with parameters): + + A. non-qualified: + + 1) specifying parameters names: + + `foo(a, b)`_ + + 2) specifying parameters types: + + `foo(int, float)`_ + + 3) specifying both names and types: + + `foo(a: int, b: float)`_ + + 4) output parameter can also be specified if you wish: + + `foo(a: int, b: float): string`_ + + B. qualified: all 4 options above are valid. + Particularly you can use the full format: + + `proc foo(a: int, b: float): string`_ + +.. Tip:: Avoid cluttering your text with extraneous information by using + one of shorter forms: + + binarySearch_ + `binarySearch(a, key, cmp)`_ + + Brevity is better for reading! If you use a short form and have an + ambiguity problem (see below) then just add some additional info. + +Symbol kind like `proc` can also be specified in the postfix form: + + `foo proc`_ + `walkDir(d: string) iterator`_ + +.. Warning:: An ambiguity in resolving documentation links may arise because of: + + 1. clash with other RST anchors + * manually setup anchors + * automatically set up, e.g. section names + 2. collision with other Nim symbols: + + * routines with different parameters can exist e.g. for + `proc` and `template`. In this case they are split between their + corresponding sections in output file. Qualified references are + useful in this case -- just disambiguate by referring to these + sections explicitly: + + See `foo proc`_ and `foo template`_. + + * because in Nim `proc` and `iterator` belong to different namespaces, + so there can be a collision even if parameters are the same. + Use `\`proc foo\`_`:literal: or `\`iterator foo\`_`:literal: then. + + Any ambiguity is always reported with Nim compiler warnings and an anchor + with higher priority is selected. Manual anchors have highest + priority, then go automatic RST anchors; then Nim-generated anchors + (while procs have higher priority than other Nim symbol kinds). + +Generic parameters can also be used. All in all, this long form will be +recognized fine: + + `proc binarySearch*[T; K](a: openArray[T], key: K, cmp: proc(T, K)): int`_ + +**Limitations**: + +1. The parameters of a nested routine type can be specified only with types + (without parameter names, see form A.2 above). + E.g. for this signature: + + ```Nim + proc binarySearch*[T, K](a: openArray[T]; key: K; + cmp: proc (x: T; y: K): int {.closure.}): int + ~~ ~~ ~~~~~ + ``` + + you cannot use names underlined by `~~` so it must be referenced with + ``cmp: proc(T, K)``. Hence these forms are valid: + + `binarySearch(a: openArray[T], key: K, cmp: proc(T, K))`_ + `binarySearch(openArray[T], K, proc(T, K))`_ + `binarySearch(a, key, cmp)`_ +2. Default values in routine parameters are not recognized, one needs to + specify the type and/or name instead. E.g. for referencing `proc f(x = 7)` + use one of the mentioned forms: + + `f(int)`_ or `f(x)`_ or `f(x: int)`_. +3. Generic parameters must be given the same way as in the + definition of referenced symbol. + + * their names should be the same + * parameters list should be given the same way, e.g. without substitutions + between commas (,) and semicolons (;). + +.. Note:: A bit special case is operators + (as their signature is also defined with `\``): + + ```Nim + func `$`(x: MyType): string + func `[]`*[T](x: openArray[T]): T + ``` + + A short form works without additional backticks: + + `$`_ + `[]`_ + + However for fully-qualified reference copy-pasting backticks (`) into other + backticks will not work in our RST parser (because we use Markdown-like + inline markup rules). You need either to delete backticks or keep + them and escape with backslash \\: + + no backticks: `func $`_ + escaped: `func \`$\``_ + no backticks: `func [][T](x: openArray[T]): T`_ + escaped: `func \`[]\`[T](x: openArray[T]): T`_ + +.. Note:: Types that defined as `enum`, or `object`, or `tuple` can also be + referenced with those names directly (instead of `type`): + + type CopyFlag = enum + ... + ## Ref. `CopyFlag enum`_ + +Nim external referencing +------------------------ + +Just like for [Markup external referencing], which saves markup anchors, +the Nim symbols are also saved in ``.idx`` files, so one needs +to generate them beforehand, and they should be loaded by +an ``.. importdoc::`` directive. Arguments to ``.. importdoc::`` is a +comma-separated list of Nim modules or Markdown/RST documents. + +`--index:only`:option: tells Nim to only generate ``.idx`` file and +do **not** attempt to generate HTML/LaTeX output. +For ``.nim`` modules there are 2 alternatives to work with ``.idx`` files: + +1. using [Project switch] implies generation of ``.idx`` files, + however, if ``importdoc`` is called on upper modules as its arguments, + their ``.idx`` are not yet created. Thus one should generate **all** + required ``.idx`` first: + ```cmd + nim doc --project --index:only <main>.nim + nim doc --project <main>.nim + ``` +2. or run `nim doc --index:only <module.nim>`:cmd: command for **all** (used) + Nim modules in your project. Then run `nim doc <module.nim>` on them for + output HTML generation. + + .. Warning:: A mere `nim doc --index:on`:cmd: may fail on an attempt to do + ``importdoc`` from another module (for which ``.idx`` was not yet + generated), that's why `--index:only`:option: shall be used instead. + + For ``.md``/``.rst`` markup documents point 2 is the only option. + +Then, you can freely use something like this in ``your_module.nim``: + + ```nim + ## .. importdoc:: user_manual.md, another_module.nim + + ... + ## Ref. [some section from User Manual]. + + ... + ## Ref. [proc f] + ## (assuming you have a proc `f` in ``another_module``). + ``` + +and compile it by `nim doc`:cmd:. Note that link text will +be automatically prefixed by the module name of symbol, +so you will see something like "Ref. [another_module: proc f](#)" +in the generated output. + +It's also possible to reference a whole module by prefixing or +suffixing full canonical module name with "module": + + Ref. [module subdir/name] or [subdir/name module]. + +Markup documents as a whole can be referenced just by their title +(or by their file name if the title was not set) without any prefix. + +.. Tip:: During development process the stage of ``.idx`` files generation + can be done only *once*, after that you use already generated ``.idx`` + files while working with a document *being developed* (unless you do + incompatible changes to *referenced* documents). + +.. Hint:: After changing a *referenced* document file one may need + to regenerate its corresponding ``.idx`` file to get correct results. + Of course, when referencing *internally* inside any given ``.nim`` file, + it's not needed, one can even immediately use any freshly added anchor + (a document's own ``.idx`` file is not used for resolving its internal links). + +If an ``importdoc`` directive fails to find a ``.idx``, then an error +is emitted. + +In case of such compilation failures please note that: + +* **all** relative paths, given to ``importdoc``, relate to insides of + ``OUTDIR``, and **not** project's directory structure. + +* ``importdoc`` searches for ``.idx`` in `--outdir:OUTDIR`:option: directory + (``htmldocs`` by default) and **not** around original modules, so: + + .. Tip:: look into ``OUTDIR`` to understand what's going on. + +* also keep in mind that ``.html`` and ``.idx`` files should always be + output to the same directory, so check this and, if it's not true, check + that both runs *with* and *without* `--index:only`:option: have all + other options the same. + +To summarize, for 2 basic options of [Structuring output directories] +compilation options are different: + +1) complex hierarchy with `--project`:option: switch. + + As the **original** project's directory structure is replicated in + `OUTDIR`, all passed paths are related to this structure also. + + E.g. if a module ``path1/module.nim`` does + ``.. importdoc:: path2/another.nim`` then docgen tries to load file + ``OUTDIR/path1/path2/another.idx``. + + .. Note:: markup documents are just placed into the specified directory + `OUTDIR`:option: by default (i.e. they are **not** affected by + `--project`:option:), so if you have ``PROJECT/doc/manual.md`` + document and want to use complex hierarchy (with ``doc/``), + compile it with `--docroot`:option:\: + ```cmd + # 1st stage + nim md2html --outdir:OUTDIR --docroot:/absolute/path/to/PROJECT \ + --index:only PROJECT/doc/manual.md + ... + # 2nd stage + nim md2html --outdir:OUTDIR --docroot:/absolute/path/to/PROJECT \ + PROJECT/doc/manual.md + ``` + + Then the output file will be placed as ``OUTDIR/doc/manual.idx``. + So if you have ``PROJECT/path1/module.nim``, then ``manual.md`` can + be referenced as ``../doc/manual.md``. + +2) flattened structure. + + E.g. if a module ``path1/module.nim`` does + ``.. importdoc:: path2/another.nim`` then docgen tries to load + ``OUTDIR/path2/another.idx``, so the path ``path1`` + does not matter and providing ``path2`` can be useful only + in the case it contains another package that was placed there + using `--outdir:OUTDIR/path2`:option:. + + The links' text will be prefixed as ``another: ...`` in both cases. + + .. Warning:: Again, the same `--outdir:OUTDIR`:option: option should + be provided to both `doc --index:only`:option: / + `md2html --index:only`:option: and final generation by + `doc`:option:/`md2html`:option: inside 1 package. + +To temporarily disable ``importdoc``, e.g. if you don't need +correct link resolution at the moment, use a `--noImportdoc`:option: switch +(only warnings about unresolved links will be generated for external references). + +Related Options +=============== + +Project switch +-------------- + + ```cmd + nim doc --project filename.nim + ``` + +This will recursively generate documentation of all Nim modules imported +into the input module that belong to the Nimble package that ``filename.nim`` +belongs to. The index files and the corresponding ``theindex.html`` will +also be generated. + + +Index switch +------------ + + ```cmd + nim doc --index:on filename.nim + ``` + +This will generate an index of all the exported symbols in the input Nim +module, and put it into a neighboring file with the extension of ``.idx``. The +index file is line-oriented (newlines have to be escaped). Each line +represents a tab-separated record of several columns, the first two mandatory, +the rest optional. See the [Index (idx) file format] section for details. + +.. Note:: `--index`:option: switch only affects creation of ``.idx`` + index files, while user-searchable Index HTML file is created by + `buildIndex`:option: command. + +Buildindex command +------------------ + +Once index files have been generated for one or more modules, the Nim +compiler command `nim buildIndex directory`:cmd: can be run to go over all the index +files in the specified directory to generate a [theindex.html](theindex.html) +file: + + ```cmd + nim buildIndex -o:path/to/htmldocs/theindex.html path/to/htmldocs + ``` + +See source switch +----------------- + + ```cmd + nim doc --git.url:<url> filename.nim + ``` + +With the `git.url`:option: switch the *See source* hyperlink will appear below each +documented item in your source code pointing to the implementation of that +item on a GitHub repository. +You can click the link to see the implementation of the item. + +The `git.commit`:option: switch overrides the hardcoded `devel` branch in +``config/nimdoc.cfg``. +This is useful to link to a different branch e.g. `--git.commit:master`:option:, +or to a tag e.g. `--git.commit:1.2.3`:option: or a commit. + +Source URLs are generated as ``href="${url}/tree/${commit}/${path}#L${line}"`` +by default and thus compatible with GitHub but not with GitLab. + +Similarly, `git.devel`:option: switch overrides the hardcoded `devel` branch +for the `Edit` link which is also useful if you have a different working +branch than `devel` e.g. `--git.devel:master`:option:. + +Edit URLs are generated as ``href="${url}/tree/${devel}/${path}#L${line}"`` +by default. + +You can edit ``config/nimdoc.cfg`` and modify the ``doc.item.seesrc`` value +with a hyperlink to your own code repository. + +In the case of Nim's own documentation, the `commit` value is just a commit +hash to append to a formatted URL to https://github.com/nim-lang/Nim. + + +Other Input Formats +=================== + +The *Nim compiler* also has support for RST (reStructuredText) files with +the `rst2html`:option: and `rst2tex`:option: commands. Documents like this one are +initially written in a dialect of RST which adds support for Nim source code +highlighting with the ``.. code-block:: nim`` prefix. ``code-block`` also +supports highlighting of a few other languages supported by the +[packages/docutils/highlite module](highlite.html). + +See [Markdown and RST markup languages](markdown_rst.html) for +usage of those commands. + +HTML anchor generation +====================== + +When you run the `rst2html`:option: command, all sections in the RST document will +get an anchor you can hyperlink to. Usually, you can guess the anchor lower +casing the section title and replacing spaces with dashes, and in any case, you +can get it from the table of contents. But when you run the `doc`:option: +command to generate API documentation, some symbol get one or two anchors at +the same time: a numerical identifier, or a plain name plus a complex name. + +The numerical identifier is just a random number. The number gets assigned +according to the section and position of the symbol in the file being processed +and you should not rely on it being constant: if you add or remove a symbol the +numbers may shuffle around. + +The plain name of a symbol is a simplified version of its fully exported +signature. Variables or constants have the same plain name symbol as their +complex name. The plain name for procs, templates, and other callable types +will be their unquoted value after removing parameters, return types, and +pragmas. The plain name allows short and nice linking of symbols that works +unless you have a module with collisions due to overloading. + +If you hyperlink a plain name symbol and there are other matches on the same +HTML file, most browsers will go to the first one. To differentiate the rest, +you will need to use the complex name. A complex name for a callable type is +made up of several parts: + + (**plain symbol**)(**.type**),(**first param**)?(**,param type**)\* + +The first thing to note is that all callable types have at least a comma, even +if they don't have any parameters. If there are parameters, they are +represented by their types and will be comma-separated. To the plain symbol a +suffix may be added depending on the type of the callable: + +============== ============== +Callable type Suffix +============== ============== +`proc`, `func` *empty string* +`macro` ``.m`` +`method` ``.e`` +`iterator` ``.i`` +`template` ``.t`` +`converter` ``.c`` +============== ============== + +The relationship of type to suffix is made by the proc `complexName` in the +``compiler/docgen.nim`` file. Here are some examples of complex names for +symbols in the [system module](system.html). + +* `type SomeSignedInt = int | int8 | int16 | int32 | int64` **=>** + [#SomeSignedInt](system.html#SomeSignedInt) +* `var globalRaiseHook: proc (e: ref E_Base): bool {.nimcall.}` **=>** + [#globalRaiseHook](system.html#globalRaiseHook) +* `const NimVersion = "0.0.0"` **=>** + [#NimVersion](system.html#NimVersion) +* `proc getTotalMem(): int {.rtl, raises: [], tags: [].}` **=>** + [#getTotalMem](system.html#getTotalMem) +* `proc len[T](x: seq[T]): int {.magic: "LengthSeq", noSideEffect.}` **=>** + [#len,seq[T]](system.html#len,seq[T]) +* `iterator pairs[T](a: seq[T]): tuple[key: int, val: T] {.inline.}` **=>** + [#pairs.i,seq[T]](iterators.html#pairs.i,seq[T]) +* `template newException[](exceptn: typedesc; message: string; + parentException: ref Exception = nil): untyped` **=>** + [#newException.t,typedesc,string,ref.Exception]( + system.html#newException.t,typedesc,string,ref.Exception) + + +Index (idx) file format +======================= + +Files with the ``.idx`` extension are generated when you use the [Index +switch] along with commands to generate +documentation from source or text files. You can programmatically generate +indices with the [setIndexTerm()]( +rstgen.html#setIndexTerm,RstGenerator,string,string,string,string,string) +and `writeIndexFile() <rstgen.html#writeIndexFile,RstGenerator,string>`_ procs. +The purpose of `idx` files is to hold the interesting symbols and their HTML +references so they can be later concatenated into a big index file with +[mergeIndexes()](rstgen.html#mergeIndexes,string). This section documents +the file format in detail. + +Index files are line-oriented and tab-separated (newline and tab characters +have to be escaped). Each line represents a record with 6 fields. +The content of these columns is: + +0. Discriminator tag denoting type of the index entry, allowed values are: + `markupTitle` + : a title for ``.md``/``.rst`` document + `nimTitle` + : a title of ``.nim`` module + `heading` + : heading of sections, can be both in Nim and markup files + `idx` + : terms marked with :idx: role + `nim` + : a Nim symbol + `nimgrp` + : a Nim group for overloadable symbols like `proc`s +1. Mandatory term being indexed. Terms can include quoting according to + Nim's rules (e.g. \`^\`). +2. Base filename plus anchor hyperlink (e.g. ``algorithm.html#*,int,SortOrder``). +3. Optional human-readable string to display as a hyperlink. If the value is not + present or is the empty string, the hyperlink will be rendered + using the term. Prefix whitespace indicates that this entry is + not for an API symbol but for a TOC entry. +4. Optional title or description of the hyperlink. Browsers usually display + this as a tooltip after hovering a moment over the hyperlink. +5. A line number of file where the entry was defined. + +The index generation tools differentiate between documentation +generated from ``.nim`` files and documentation generated from ``.md`` or +``.rst`` files by tag `nimTitle` or `markupTitle` in the 1st line of +the ``.idx`` file. + +.. TODO Normal symbols are added to the index with surrounding whitespaces removed. An + exception to this are the table of content (TOC) entries. TOC entries are added to + the index file with their third column having as much prefix spaces as their + level is in the TOC (at least 1 character). The prefix whitespace helps to + filter TOC entries from API or text symbols. This is important because the + amount of spaces is used to replicate the hierarchy for document TOCs in the + final index, and TOC entries found in ``.nim`` files are discarded. + + +Additional resources +==================== + +* [Nim Compiler User Guide](nimc.html#compiler-usage-commandminusline-switches) + +* already mentioned documentation for + [Markdown and RST markup languages](markdown_rst.html), which also + contains the list of implemented features of these markup languages. + +* the implementation is in [module compiler/docgen]. + +The output for HTML and LaTeX comes from the ``config/nimdoc.cfg`` and +``config/nimdoc.tex.cfg`` configuration files. You can add and modify these +files to your project to change the look of the docgen output. + +You can import the [packages/docutils/rstgen module](rstgen.html) in your +programs if you want to reuse the compiler's documentation generation procs. diff --git a/doc/docgen.rst b/doc/docgen.rst deleted file mode 100644 index dac8808ad..000000000 --- a/doc/docgen.rst +++ /dev/null @@ -1,395 +0,0 @@ -=================================== - Nim DocGen Tools Guide -=================================== - -:Author: Erik O'Leary -:Version: |nimversion| - -.. contents:: - - -Introduction -============ - -This document describes the `documentation generation tools`:idx: built into -the `Nim compiler <nimc.html>`_, which can generate HTML and JSON output -from input .nim files and projects, as well as HTML and LaTeX from input RST -(reStructuredText) files. The output documentation will include module -dependencies (``import``), any top-level documentation comments (##), and -exported symbols (*), including procedures, types, and variables. - -Quick start ------------ - -Generate HTML documentation for a file: - -:: - nim doc <filename>.nim - -Generate HTML documentation for a whole project: - -:: - # delete any htmldocs/*.idx file before starting - nim doc --project --index:on --git.url:<url> --git.commit:<tag> --outdir:htmldocs <main_filename>.nim - # this will generate html files, a theindex.html index, css and js under `htmldocs` - # See also `--docroot` to specify a relative root. - # to get search (dochacks.js) to work locally, you need a server otherwise - # CORS will prevent opening file:// urls; this works: - python3 -m http.server 7029 --directory htmldocs - # When --outdir is omitted it defaults to $projectPath/htmldocs, - or `$nimcache/htmldocs` with `--usenimcache` which avoids clobbering your sources; - and likewise without `--project`. - Adding `-r` will open in a browser directly. - - -Documentation Comments ----------------------- - -Any comments which are preceded by a double-hash (##), are interpreted as -documentation. Comments are parsed as RST (see `reference -<http://docutils.sourceforge.net/docs/user/rst/quickref.html>`_), providing -Nim module authors the ability to easily generate richly formatted -documentation with only their well-documented code. - -Example: - -.. code-block:: nim - type Person* = object - ## This type contains a description of a person - name: string - age: int - -Outputs:: - Person* = object - name: string - age: int - -This type contains a description of a person - -Field documentation comments can be added to fields like so: - -.. code-block:: nim - var numValues: int ## \ - ## `numValues` stores the number of values - -Note that without the `*` following the name of the type, the documentation for -this type would not be generated. Documentation will only be generated for -*exported* types/procedures/etc. - - -Nim file input ------------------ - -The following examples will generate documentation for the below contrived -*Nim* module, aptly named 'sample.nim' - -sample.nim: - -.. code-block:: nim - ## This module is a sample. - - import strutils - - proc helloWorld*(times: int) = - ## Takes an integer and outputs - ## as many "hello world!"s - - for i in 0 .. times-1: - echo "hello world!" - - helloWorld(5) - - -Document Types -============== - - -HTML ----- - -Generation of HTML documents is done via the ``doc`` command. This command -takes either a single .nim file, outputting a single .html file with the same -base filename, or multiple .nim files, outputting multiple .html files and, -optionally, an index file. - -The ``doc`` command:: - nim doc sample - -Partial Output:: - ... - proc helloWorld(times: int) {.raises: [], tags: [].} - ... - -The full output can be seen here: `docgen_sample.html <docgen_sample.html>`_. -It runs after semantic checking, and includes pragmas attached implicitly by the -compiler. - - -JSON ----- - -Generation of JSON documents is done via the ``jsondoc`` command. This command -takes in a .nim file, and outputs a .json file with the same base filename. Note -that this tool is built off of the ``doc`` command (previously ``doc2``), and -contains the same information. - -The ``jsondoc`` command:: - nim jsondoc sample - -Output:: - { - "orig": "docgen_sample.nim", - "nimble": "", - "moduleDescription": "This module is a sample", - "entries": [ - { - "name": "helloWorld", - "type": "skProc", - "line": 5, - "col": 0, - "description": "Takes an integer and outputs as many "hello world!"s", - "code": "proc helloWorld(times: int) {.raises: [], tags: [].}" - } - ] - } - -Similarly to the old ``doc`` command the old ``jsondoc`` command has been -renamed ``jsondoc0``. - -The ``jsondoc0`` command:: - nim jsondoc0 sample - -Output:: - [ - { - "comment": "This module is a sample." - }, - { - "name": "helloWorld", - "type": "skProc", - "description": "Takes an integer and outputs as many "hello world!"s", - "code": "proc helloWorld*(times: int)" - } - ] - -Note that the ``jsondoc`` command outputs it's JSON without pretty-printing it, -while ``jsondoc0`` outputs pretty-printed JSON. - -Related Options -=============== - -Project switch --------------- - -:: - nim doc --project filename.nim - -This will recursively generate documentation of all nim modules imported -into the input module that belong to the Nimble package that ``filename.nim`` -belongs to. - - -Index switch ------------- - -:: - nim doc2 --index:on filename.nim - -This will generate an index of all the exported symbols in the input Nim -module, and put it into a neighboring file with the extension of ``.idx``. The -index file is line oriented (newlines have to be escaped). Each line -represents a tab separated record of several columns, the first two mandatory, -the rest optional. See the `Index (idx) file format`_ section for details. - -Once index files have been generated for one or more modules, the Nim -compiler command ``buildIndex directory`` can be run to go over all the index -files in the specified directory to generate a `theindex.html <theindex.html>`_ -file. - -See source switch ------------------ - -The ``docSeeSrcUrl`` switch is deprecated. Use: - -:: - nim doc2 --git.url:<url> filename.nim - -With the ``git.url`` switch the *See source* hyperlink will appear below each -documented item in your source code pointing to the implementation of that -item on a GitHub repository. -You can click the link to see the implementation of the item. - -The ``git.commit`` switch overrides the hardcoded `devel` branch in config/nimdoc.cfg. -This is useful to link to a different branch e.g. `--git.commit:master`, -or to a tag e.g. `--git.commit:1.2.3` or a commit. - -Source URLs are generated as `href="${url}/tree/${commit}/${path}#L${line}"` by default and this compatible with GitHub but not with GitLab. - -Similarly, ``git.devel`` switch overrides the hardcoded `devel` branch for the `Edit` link which is also useful if you have a different working branch than `devel` e.g. `--git.devel:master`. - -Edit URLs are generated as `href="${url}/tree/${devel}/${path}#L${line}"` by default. - -You can edit ``config/nimdoc.cfg`` and modify the ``doc.item.seesrc`` value with a hyperlink to your own code repository. - -In the case of Nim's own documentation, the ``commit`` value is just a commit -hash to append to a formatted URL to https://github.com/nim-lang/Nim. The -``tools/nimweb.nim`` helper queries the current git commit hash during doc -generation, but since you might be working on an unpublished repository, it -also allows specifying a ``githash`` value in ``web/website.ini`` to force a -specific commit in the output. - - -Other Input Formats -=================== - -The *Nim compiler* also has support for RST (reStructuredText) files with -the ``rst2html`` and ``rst2tex`` commands. Documents like this one are -initially written in a dialect of RST which adds support for nim source code -highlighting with the ``.. code-block:: nim`` prefix. ``code-block`` also -supports highlighting of C++ and some other c-like languages. - -Usage:: - nim rst2html docgen.txt - -Output:: - You're reading it! - -The ``rst2tex`` command is invoked identically to ``rst2html``, but outputs -a .tex file instead of .html. - - -HTML anchor generation -====================== - -When you run the ``rst2html`` command, all sections in the RST document will -get an anchor you can hyperlink to. Usually you can guess the anchor lower -casing the section title and replacing spaces with dashes, and in any case you -can get it from the table of contents. But when you run the ``doc`` or ``doc2`` -commands to generate API documentation, some symbol get one or two anchors at -the same time: a numerical identifier, or a plain name plus a complex name. - -The numerical identifier is just a random number. The number gets assigned -according to the section and position of the symbol in the file being processed -and you should not rely on it being constant: if you add or remove a symbol the -numbers may shuffle around. - -The plain name of a symbol is a simplified version of its fully exported -signature. Variables or constants have the same plain name symbol as their -complex name. The plain name for procs, templates, and other callable types -will be their unquoted value after removing parameters, return types and -pragmas. The plain name allows short and nice linking of symbols which works -unless you have a module with collisions due to overloading. - -If you hyperlink a plain name symbol and there are other matches on the same -HTML file, most browsers will go to the first one. To differentiate the rest, -you will need to use the complex name. A complex name for a callable type is -made up from several parts: - - (**plain symbol**)(**.type**),(**first param**)?(**,param type**)\* - -The first thing to note is that all callable types have at least a comma, even -if they don't have any parameters. If there are parameters, they are -represented by their types and will be comma separated. To the plain symbol a -suffix may be added depending on the type of the callable: - -------------- -------------- -Callable type Suffix -------------- -------------- -proc *empty string* -macro ``.m`` -method ``.e`` -iterator ``.i`` -template ``.t`` -converter ``.c`` -------------- -------------- - -The relationship of type to suffix is made by the proc ``complexName`` in the -``compiler/docgen.nim`` file. Here are some examples of complex names for -symbols in the `system module <system.html>`_. - -* ``type SomeSignedInt = int | int8 | int16 | int32 | int64`` **=>** - `#SomeSignedInt <system.html#SomeSignedInt>`_ -* ``var globalRaiseHook: proc (e: ref E_Base): bool {.nimcall.}`` **=>** - `#globalRaiseHook <system.html#globalRaiseHook>`_ -* ``const NimVersion = "0.0.0"`` **=>** - `#NimVersion <system.html#NimVersion>`_ -* ``proc getTotalMem(): int {.rtl, raises: [], tags: [].}`` **=>** - `#getTotalMem, <system.html#getTotalMem>`_ -* ``proc len[T](x: seq[T]): int {.magic: "LengthSeq", noSideEffect.}`` **=>** - `#len,seq[T] <system.html#len,seq[T]>`_ -* ``iterator pairs[T](a: seq[T]): tuple[key: int, val: T] {.inline.}`` **=>** - `#pairs.i,seq[T] <system.html#pairs.i,seq[T]>`_ -* ``template newException[](exceptn: typedesc; message: string; - parentException: ref Exception = nil): untyped`` **=>** - `#newException.t,typedesc,string,ref.Exception - <system.html#newException.t,typedesc,string,ref.Exception>`_ - - -Index (idx) file format -======================= - -Files with the ``.idx`` extension are generated when you use the `Index -switch <#related-options-index-switch>`_ along with commands to generate -documentation from source or text files. You can programmatically generate -indices with the `setIndexTerm() -<rstgen.html#setIndexTerm,RstGenerator,string,string,string,string,string>`_ -and `writeIndexFile() <rstgen.html#writeIndexFile,RstGenerator,string>`_ procs. -The purpose of ``idx`` files is to hold the interesting symbols and their HTML -references so they can be later concatenated into a big index file with -`mergeIndexes() <rstgen.html#mergeIndexes,string>`_. This section documents -the file format in detail. - -Index files are line oriented and tab separated (newline and tab characters -have to be escaped). Each line represents a record with at least two fields, -but can have up to four (additional columns are ignored). The content of these -columns is: - -1. Mandatory term being indexed. Terms can include quoting according to - Nim's rules (eg. \`^\`). -2. Base filename plus anchor hyperlink (eg. - ``algorithm.html#*,int,SortOrder``). -3. Optional human readable string to display as hyperlink. If the value is not - present or is the empty string, the hyperlink will be rendered - using the term. Prefix whitespace indicates that this entry is - not for an API symbol but for a TOC entry. -4. Optional title or description of the hyperlink. Browsers usually display - this as a tooltip after hovering a moment over the hyperlink. - -The index generation tools try to differentiate between documentation -generated from ``.nim`` files and documentation generated from ``.txt`` or -``.rst`` files. The former are always closely related to source code and -consist mainly of API entries. The latter are generic documents meant for -human reading. - -To differentiate both types (documents and APIs), the index generator will add -to the index of documents an entry with the title of the document. Since the -title is the topmost element, it will be added with a second field containing -just the filename without any HTML anchor. By convention this entry without -anchor is the *title entry*, and since entries in the index file are added as -they are scanned, the title entry will be the first line. The title for APIs -is not present because it can be generated concatenating the name of the file -to the word **Module**. - -Normal symbols are added to the index with surrounding whitespaces removed. An -exception to this are table of content (TOC) entries. TOC entries are added to -the index file with their third column having as much prefix spaces as their -level is in the TOC (at least 1 character). The prefix whitespace helps to -filter TOC entries from API or text symbols. This is important because the -amount of spaces is used to replicate the hierarchy for document TOCs in the -final index, and TOC entries found in ``.nim`` files are discarded. - - -Additional resources -==================== - -`Nim Compiler User Guide <nimc.html#compiler-usage-command-line-switches>`_ - -`RST Quick Reference -<http://docutils.sourceforge.net/docs/user/rst/quickref.html>`_ - -The output for HTML and LaTeX comes from the ``config/nimdoc.cfg`` and -``config/nimdoc.tex.cfg`` configuration files. You can add and modify these -files to your project to change the look of docgen output. - -You can import the `packages/docutils/rstgen module <rstgen.html>`_ in your -programs if you want to reuse the compiler's documentation generation procs. diff --git a/doc/docgen_sample.nim b/doc/docgen_sample.nim index 7a167cb45..06b8d7f8e 100644 --- a/doc/docgen_sample.nim +++ b/doc/docgen_sample.nim @@ -1,6 +1,6 @@ ## This module is a sample. -import strutils +import std/strutils proc helloWorld*(times: int) = ## Takes an integer and outputs diff --git a/doc/docs.rst b/doc/docs.md index b34383253..b6ff6d2c7 100644 --- a/doc/docs.rst +++ b/doc/docs.md @@ -1,38 +1,38 @@ The documentation consists of several documents: -- | `Tutorial (part I) <tut1.html>`_ +- | [Tutorial (part I)](tut1.html) | The Nim tutorial part one deals with the basics. -- | `Tutorial (part II) <tut2.html>`_ +- | [Tutorial (part II)](tut2.html) | The Nim tutorial part two deals with the advanced language constructs. -- | `Tutorial (part III) <tut3.html>`_ +- | [Tutorial (part III)](tut3.html) | The Nim tutorial part three about Nim's macro system. -- | `Language Manual <manual.html>`_ +- | [Language Manual](manual.html) | The Nim manual is a draft that will evolve into a proper specification. -- | `Library documentation <lib.html>`_ +- | [Library documentation](lib.html) | This document describes Nim's standard library. -- | `Compiler user guide <nimc.html>`_ +- | [Compiler user guide](nimc.html) | The user guide lists command line arguments, special features of the compiler, etc. -- | `Tools documentation <tools.html>`_ +- | [Tools documentation](tools.html) | Description of some tools that come with the standard distribution. -- | `GC <gc.html>`_ - | Additional documentation about Nim's multi-paradigm memory management strategies +- | [Memory management](mm.html) + | Additional documentation about Nim's memory management strategies | and how to operate them in a realtime setting. -- | `Source code filters <filters.html>`_ +- | [Source code filters](filters.html) | The Nim compiler supports source code filters as a simple yet powerful builtin templating system. -- | `Internal documentation <intern.html>`_ +- | [Internal documentation](intern.html) | The internal documentation describes how the compiler is implemented. Read this if you want to hack the compiler. -- | `Index <theindex.html>`_ +- | [Index](theindex.html) | The generated index. **Index + (Ctrl+F) == Joy** diff --git a/doc/docstyle.rst b/doc/docstyle.md index 70ded0689..291a34cf6 100644 --- a/doc/docstyle.rst +++ b/doc/docstyle.md @@ -4,49 +4,52 @@ Documentation Style General Guidelines ------------------ -* See also `nep1<https://nim-lang.github.io/Nim/nep1.html>`_ which should probably be merged here. +* See also [nep1](nep1.html) which should probably be merged here. * Authors should document anything that is exported; documentation for private - procs can be useful too (visible via ``nim doc --docInternal foo.nim``). + procs can be useful too (visible via `nim doc --docInternal foo.nim`:cmd:). * Within documentation, a period (`.`) should follow each sentence (or sentence fragment) in a comment block. The documentation may be limited to one sentence fragment, but if multiple sentences are within the documentation, each sentence after the first should be complete and in present tense. * Documentation is parsed as a custom ReStructuredText (RST) with partial markdown support. * In nim sources, prefer single backticks to double backticks since it's simpler - and `nim doc` supports it (even in rst files with `nim rst2html`). -* In nim sources, for links, prefer `[link text](link.html)` to ``` `link text<link.html>`_ ``` - since the syntax is simpler and markdown is more common (likewise, `nim rst2html` also supports it in rst files). - -.. code-block:: nim + and `nim doc`:cmd: supports it. Likewise with ``rst`` files: `nim rst2html`:cmd: will render those as monospace, and + adding ``.. default-role:: code`` to an ``rst`` file will also make those render as monospace when rendered directly + in tools such as github. +* (debatable) In nim sources, for links, prefer ``[link text](link.html)`` to `\`link text<link.html>\`_`:code: + since the syntax is simpler and markdown is more common (likewise, `nim rst2html`:cmd: also supports it in ``rst`` files). + ```nim proc someproc*(s: string, foo: int) = - ## Use single backticks for inline code, eg: `s` or `someExpr(true)`. + ## Use single backticks for inline code, e.g.: `s` or `someExpr(true)`. ## Use a backlash to follow with alphanumeric char: `int8`\s are great. + ``` Module-level documentation -------------------------- -Documentation of a module is placed at the top of the module itself. Each line of documentation begins with double hashes (``##``). -Sometimes ``##[ multiline docs containing code ]##`` is preferable, see ``lib/pure/times.nim``. +Documentation of a module is placed at the top of the module itself. Each line of documentation begins with double hashes (`##`). +Sometimes `##[ multiline docs containing code ]##` is preferable, see ``lib/pure/times.nim``. Code samples are encouraged, and should follow the general RST syntax: -.. code-block:: Nim - + ````Nim ## The `universe` module computes the answer to life, the universe, and everything. ## - ## .. code-block:: - ## doAssert computeAnswerString() == 42 + ## ``` + ## doAssert computeAnswerString() == 42 + ## ``` + ```` Within this top-level comment, you can indicate the authorship and copyright of the code, which will be featured in the produced documentation. -.. code-block:: Nim - + ```Nim ## This is the best module ever. It provides answers to everything! ## ## :Author: Steve McQueen ## :Copyright: 1965 ## + ``` Leave a space between the last line of top-level documentation and the beginning of Nim code (the imports, etc.). @@ -55,38 +58,38 @@ Procs, Templates, Macros, Converters, and Iterators The documentation of a procedure should begin with a capital letter and should be in present tense. Variables referenced in the documentation should be surrounded by single tick marks: -.. code-block:: Nim - + ```Nim proc example1*(x: int) = ## Prints the value of `x`. echo x + ``` Whenever an example of usage would be helpful to the user, you should include one within the documentation in RST format as below. -.. code-block:: Nim - + ````Nim proc addThree*(x, y, z: int8): int = ## Adds three `int8` values, treating them as unsigned and ## truncating the result. ## - ## .. code-block:: - ## # things that aren't suitable for a `runnableExamples` go in code-block: - ## echo execCmdEx("git pull") - ## drawOnScreen() + ## ``` + ## # things that aren't suitable for a `runnableExamples` go in code block: + ## echo execCmdEx("git pull") + ## drawOnScreen() + ## ``` runnableExamples: - # `runnableExamples` is usually preferred to `code-block`, when possible. + # `runnableExamples` is usually preferred to code blocks, when possible. doAssert addThree(3, 125, 6) == -122 result = x +% y +% z + ```` -The commands ``nim doc`` and ``nim doc2`` will then correctly syntax highlight the Nim code within the documentation. +The command `nim doc`:cmd: will then correctly syntax highlight the Nim code within the documentation. Types ----- Exported types should also be documented. This documentation can also contain code samples, but those are better placed with the functions to which they refer. -.. code-block:: Nim - + ```Nim type NamedQueue*[T] = object ## Provides a linked data structure with names ## throughout. It is named for convenience. I'm making @@ -94,12 +97,12 @@ Exported types should also be documented. This documentation can also contain co name*: string ## The name of the item val*: T ## Its value next*: ref NamedQueue[T] ## The next item in the queue + ``` You have some flexibility when placing the documentation: -.. code-block:: Nim - + ```Nim type NamedQueue*[T] = object ## Provides a linked data structure with names @@ -108,53 +111,54 @@ You have some flexibility when placing the documentation: name*: string ## The name of the item val*: T ## Its value next*: ref NamedQueue[T] ## The next item in the queue + ``` Make sure to place the documentation beside or within the object. -.. code-block:: Nim - + ```Nim type - ## Bad: this documentation disappears because it annotates the ``type`` keyword - ## above, not ``NamedQueue``. + ## Bad: this documentation disappears because it annotates the `type` keyword + ## above, not `NamedQueue`. NamedQueue*[T] = object name*: string ## This becomes the main documentation for the object, which ## is not what we want. val*: T ## Its value next*: ref NamedQueue[T] ## The next item in the queue + ``` Var, Let, and Const ------------------- -When declaring module-wide constants and values, documentation is encouraged. The placement of doc comments is similar to the ``type`` sections. - -.. code-block:: Nim +When declaring module-wide constants and values, documentation is encouraged. The placement of doc comments is similar to the `type` sections. + ```Nim const X* = 42 ## An awesome number. SpreadArray* = [ [1,2,3], [2,3,1], [3,1,2], - ] ## Doc comment for ``SpreadArray``. + ] ## Doc comment for `SpreadArray`. + ``` -Placement of comments in other areas is usually allowed, but will not become part of the documentation output and should therefore be prefaced by a single hash (``#``). - -.. code-block:: Nim +Placement of comments in other areas is usually allowed, but will not become part of the documentation output and should therefore be prefaced by a single hash (`#`). + ```Nim const BadMathVals* = [ 3.14, # pi 2.72, # e 0.58, # gamma ] ## A bunch of badly rounded values. + ``` Nim supports Unicode in comments, so the above can be replaced with the following: -.. code-block:: Nim - + ```Nim const BadMathVals* = [ 3.14, # π 2.72, # e 0.58, # γ ] ## A bunch of badly rounded values (including π!). + ``` diff --git a/doc/drnim.rst b/doc/drnim.md index 5351daac9..1dc2b550f 100644 --- a/doc/drnim.rst +++ b/doc/drnim.md @@ -5,6 +5,8 @@ :Author: Andreas Rumpf :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: @@ -12,28 +14,28 @@ Introduction ============ This document describes the usage of the *DrNim* tool. DrNim combines -the Nim frontend with the `Z3 <https://github.com/Z3Prover/z3>`_ proof -engine in order to allow verify / validate software written in Nim. -DrNim's command line options are the same as the Nim compiler's. +the Nim frontend with the [Z3](https://github.com/Z3Prover/z3) proof +engine, in order to allow verify/validate software written in Nim. +DrNim's command-line options are the same as the Nim compiler's. DrNim currently only checks the sections of your code that are marked -via ``staticBoundChecks: on``: - -.. code-block:: nim +via `staticBoundChecks: on`: + ```nim {.push staticBoundChecks: on.} # <--- code section here ----> {.pop.} + ``` DrNim currently only tries to prove array indexing or subrange checks, overflow errors are *not* prevented. Overflows will be checked for in the future. Later versions of the **Nim compiler** will **assume** that the checks inside -the ``staticBoundChecks: on`` environment have been proven correct and so +the `staticBoundChecks: on` environment have been proven correct and so it will **omit** the runtime checks. If you do not want this behavior, use -instead ``{.push staticBoundChecks: defined(nimDrNim).}``. This way the +instead `{.push staticBoundChecks: defined(nimDrNim).}`. This way the Nim compiler remains unaware of the performed proofs but DrNim will prove your code. @@ -41,7 +43,8 @@ your code. Installation ============ -Run ``koch drnim``, the executable will afterwards be in ``$nim/bin/drnim``. +Run `koch drnim`:cmd:, the executable will afterwards be +in ``$nim/bin/drnim``. Motivating Example @@ -50,8 +53,7 @@ Motivating Example The follow example highlights what DrNim can easily do, even without additional annotations: -.. code-block:: nim - + ```nim {.push staticBoundChecks: on.} proc sum(a: openArray[int]): int = @@ -61,13 +63,14 @@ without additional annotations: {.pop.} echo sum([1, 2, 3]) + ``` This program contains a famous "index out of bounds" bug. DrNim -detects it and produces the following error message:: +detects it and produces the following error message: - cannot prove: i <= len(a) + -1; counter example: i -> 0 a.len -> 0 [IndexCheck] + cannot prove: i <= len(a) + -1; counter example: i -> 0 a.len -> 0 [IndexCheck] -In other words for ``i == 0`` and ``a.len == 0`` (for example!) there would be +In other words for `i == 0` and `a.len == 0` (for example!) there would be an index out of bounds error. @@ -82,37 +85,37 @@ DrNim adds 4 additional annotations (pragmas) to Nim: - `assume`:idx: These pragmas are ignored by the Nim compiler so that they don't have to -be disabled via ``when defined(nimDrNim)``. +be disabled via `when defined(nimDrNim)`. Invariant --------- -An ``invariant`` is a proposition that must be true after every loop +An `invariant` is a proposition that must be true after every loop iteration, it's tied to the loop body it's part of. Requires -------- -A ``requires`` annotation describes what the function expects to be true -before it's called so that it can perform its operation. A ``requires`` +A `requires` annotation describes what the function expects to be true +before it's called so that it can perform its operation. A `requires` annotation is also called a `precondition`:idx:. Ensures ------- -An ``ensures`` annotation describes what will be true after the function -call. An ``ensures`` annotation is also called a `postcondition`:idx:. +An `ensures` annotation describes what will be true after the function +call. An `ensures` annotation is also called a `postcondition`:idx:. Assume ------ -An ``assume`` annotation describes what DrNim should **assume** to be true +An `assume` annotation describes what DrNim should **assume** to be true in this section of the program. It is an unsafe escape mechanism comparable -to Nim's ``cast`` statement. Use it only when you really know better +to Nim's `cast` statement. Use it only when you really know better than DrNim. You should add a comment to a paper that proves the proposition you assume. @@ -122,8 +125,7 @@ Example: insertionSort **Note**: This example does not yet work with DrNim. -.. code-block:: nim - + ```nim import std / logic proc insertionSort(a: var openArray[int]) {. @@ -139,64 +141,65 @@ Example: insertionSort {.invariant: forall(j in 1..k, i in 0..<j, j == t or a[i] <= a[j]).} swap a[t], a[t-1] dec t + ``` -Unfortunately the invariants required to prove this code correct take more -code than the imperative instructions. However this effort can be compensated +Unfortunately, the invariants required to prove that this code is correct take more +code than the imperative instructions. However, this effort can be compensated by the fact that the result needs very little testing. Be aware though that -DrNim only proves that after ``insertionSort`` this condition holds:: +DrNim only proves that after `insertionSort` this condition holds: - forall(i in 1..<a.len, a[i-1] <= a[i]) + forall(i in 1..<a.len, a[i-1] <= a[i]) -This is required, but not sufficient to describe that a ``sort`` operation +This is required, but not sufficient to describe that a `sort` operation was performed. For example, the same postcondition is true for this proc which doesn't sort at all: -.. code-block:: nim - + ```nim import std / logic proc insertionSort(a: var openArray[int]) {. ensures: forall(i in 1..<a.len, a[i-1] <= a[i]).} = # does not sort, overwrites `a`'s contents! for i in 0..<a.len: a[i] = i + ``` Syntax of propositions ====================== -The basic syntax is ``ensures|requires|invariant: <prop>``. -A ``prop`` is either a comparison or a compound:: +The basic syntax is `ensures|requires|invariant: <prop>`. +A `prop` is either a comparison or a compound: - prop = nim_bool_expression - | prop 'and' prop - | prop 'or' prop - | prop '->' prop # implication - | prop '<->' prop - | 'not' prop - | '(' prop ')' # you can group props via () - | forallProp - | existsProp + prop = nim_bool_expression + | prop 'and' prop + | prop 'or' prop + | prop '->' prop # implication + | prop '<->' prop + | 'not' prop + | '(' prop ')' # you can group props via () + | forallProp + | existsProp - forallProp = 'forall' '(' quantifierList ',' prop ')' - existsProp = 'exists' '(' quantifierList ',' prop ')' + forallProp = 'forall' '(' quantifierList ',' prop ')' + existsProp = 'exists' '(' quantifierList ',' prop ')' - quantifierList = quantifier (',' quantifier)* - quantifier = <new identifier> 'in' nim_iteration_expression + quantifierList = quantifier (',' quantifier)* + quantifier = <new identifier> 'in' nim_iteration_expression -``nim_iteration_expression`` here is an ordinary expression of Nim code -that describes an iteration space, for example ``1..4`` or ``1..<a.len``. +`nim_iteration_expression` here is an ordinary expression of Nim code +that describes an iteration space, for example `1..4` or `1..<a.len`. -``nim_bool_expression`` here is an ordinary expression of Nim code of -type ``bool`` like ``a == 3`` or ``23 > a.len``. +`nim_bool_expression` here is an ordinary expression of Nim code of +type `bool` like `a == 3` or `23 > a.len`. The supported subset of Nim code that can be used in these expressions -is currently underspecified but ``let`` variables, function parameters -and ``result`` (which represents the function's final result) are amenable +is currently underspecified but `let` variables, function parameters +and `result` (which represents the function's final result) are amenable for verification. The expressions must not have any side-effects and must terminate. -The operators ``forall``, ``exists``, ``->``, ``<->`` have to imported -from ``std / logic``. +The operators `forall`, `exists`, `->`, `<->` have to imported +from `std / logic`. diff --git a/doc/effects.txt b/doc/effects.txt index 4ed1d09f1..df624e8b0 100644 --- a/doc/effects.txt +++ b/doc/effects.txt @@ -11,13 +11,14 @@ Iff a proc is side effect free and all its argument are evaluable at compile time, it can be evaluated by the compiler. However, really difficult is the ``newString`` proc: If it is simply wrapped, it should not be evaluated at compile time! On other occasions it can -and should be evaluted: +and should be evaluated: -.. code-block:: nim + ```nim proc toUpper(s: string): string = result = newString(len(s)) for i in 0..len(s) - 1: result[i] = toUpper(s[i]) + ``` No, it really can always be evaluated. The code generator should transform ``s = "\0\0\0..."`` back into ``s = newString(...)``. diff --git a/doc/estp.md b/doc/estp.md new file mode 100644 index 000000000..8a986bdf3 --- /dev/null +++ b/doc/estp.md @@ -0,0 +1,206 @@ +=================================================== + Embedded Stack Trace Profiler (ESTP) User Guide +=================================================== + +.. default-role:: code +.. include:: rstcommon.rst + +:Author: Andreas Rumpf +:Version: |nimversion| + + +Nim comes with a platform independent profiler - +the Embedded Stack Trace Profiler (ESTP). The profiler +is *embedded* into your executable. To activate the profiler you need to do: + +* compile your program with the `--profiler:on --stackTrace:on`:option: command + line options +* import the `nimprof` module +* run your program as usual. + +You can in fact look at `nimprof`'s source code to see how to implement +your own profiler. + +The setting `--profiler:on`:option: defines the conditional symbol `profiler`. +You can use `when compileOption("profiler")` to make the switch seamless. +If `profiler`:option: is `off`:option:, your program runs normally. +Otherwise your program is profiled. + +```nim +when compileOption("profiler"): + import std/nimprof +``` + +After your program has finished the profiler will create a +file ``profile_results.txt`` containing the profiling results. + +Since the profiler works by examining stack traces, it's essential that +the option `--stackTrace:on`:option: is active! Unfortunately this means that a +profiling build is much slower than a release build. + + +Memory profiler +=============== + +You can also use ESTP as a memory profiler to see which stack traces allocate +the most memory and thus create the most GC pressure. It may also help to +find memory leaks. To activate the memory profiler you need to do: + +* compile your program with the + `--profiler:off --stackTrace:on -d:memProfiler`:option: + command line options. Yes it's `--profiler:off`:option:. +* import the `nimprof` module +* run your program as usual. + +Define the symbol `ignoreAllocationSize` so that only the number of +allocations is counted and the sizes of the memory allocations do not matter. + + +Example results file +==================== + +The results file lists stack traces ordered by significance. + +The following example file has been generated by profiling the Nim compiler +itself: It shows that in total 5.4% of the runtime has been spent +in `crcFromRope` or its children. + +In general the stack traces show you immediately where the problem is because +the trace acts like an explanation; in traditional profilers you can only find +expensive leaf functions easily but the *reason* why they are invoked +often remains mysterious. + + total executions of each stack trace: + Entry: 0/3391 Calls: 84/4160 = 2.0% [sum: 84; 84/4160 = 2.0%] + newCrcFromRopeAux + crcFromRope + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 1/3391 Calls: 46/4160 = 1.1% [sum: 130; 130/4160 = 3.1%] + updateCrc32 + newCrcFromRopeAux + crcFromRope + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 2/3391 Calls: 41/4160 = 0.99% [sum: 171; 171/4160 = 4.1%] + updateCrc32 + updateCrc32 + newCrcFromRopeAux + crcFromRope + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 3/3391 Calls: 41/4160 = 0.99% [sum: 212; 212/4160 = 5.1%] + crcFromFile + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 4/3391 Calls: 41/4160 = 0.99% [sum: 253; 253/4160 = 6.1%] + updateCrc32 + crcFromFile + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 5/3391 Calls: 32/4160 = 0.77% [sum: 285; 285/4160 = 6.9%] + pop + newCrcFromRopeAux + crcFromRope + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + nim + Entry: 6/3391 Calls: 17/4160 = 0.41% [sum: 302; 302/4160 = 7.3%] + doOperation + forAllChildrenAux + pop + newCrcFromRopeAux + crcFromRope + writeRopeIfNotEqual + shouldRecompile + writeModule + myClose + closePasses + processModule + CompileModule + CompileProject + CommandCompileToC + MainCommand + HandleCmdLine + ... + nim + Entry: 7/3391 Calls: 14/4160 = 0.34% [sum: 316; 316/4160 = 7.6%] + Contains + isAccessible + interiorAllocatedPtr + gcMark + markStackAndRegisters + collectCTBody + collectCT + rawNewObj + newObj + newNode + copyTree + matchesAux + matches + resolveOverloads + semOverloadedCall + semOverloadedCallAnalyseEffects + ... + CommandCompileToC + MainCommand + HandleCmdLine diff --git a/doc/estp.rst b/doc/estp.rst deleted file mode 100644 index 805a84eb7..000000000 --- a/doc/estp.rst +++ /dev/null @@ -1,195 +0,0 @@ -=================================================== - Embedded Stack Trace Profiler (ESTP) User Guide -=================================================== - -:Author: Andreas Rumpf -:Version: |nimversion| - - -Nim comes with a platform independent profiler - -the Embedded Stack Trace Profiler (ESTP). The profiler -is *embedded* into your executable. To activate the profiler you need to do: - -* compile your program with the ``--profiler:on --stackTrace:on`` command - line options -* import the ``nimprof`` module -* run your program as usual. - -You can in fact look at ``nimprof``'s source code to see how to implement -your own profiler. - -The setting ``--profiler:on`` defines the conditional symbol ``profiler``. - -After your program has finished the profiler will create a -file ``profile_results.txt`` containing the profiling results. - -Since the profiler works by examining stack traces, it's essential that -the option ``--stackTrace:on`` is active! Unfortunately this means that a -profiling build is much slower than a release build. - - -Memory profiler -=============== - -You can also use ESTP as a memory profiler to see which stack traces allocate -the most memory and thus create the most GC pressure. It may also help to -find memory leaks. To activate the memory profiler you need to do: - -* compile your program with the ``--profiler:off --stackTrace:on -d:memProfiler`` - command line options. Yes it's ``--profiler:off``. -* import the ``nimprof`` module -* run your program as usual. - -Define the symbol ``ignoreAllocationSize`` so that only the number of -allocations is counted and the sizes of the memory allocations do not matter. - - -Example results file -==================== - -The results file lists stack traces ordered by significance. - -The following example file has been generated by profiling the Nim compiler -itself: It shows that in total 5.4% of the runtime has been spent -in ``crcFromRope`` or its children. - -In general the stack traces show you immediately where the problem is because -the trace acts like an explanation; in traditional profilers you can only find -expensive leaf functions easily but the *reason* why they are invoked -often remains mysterious. - -:: - total executions of each stack trace: - Entry: 0/3391 Calls: 84/4160 = 2.0% [sum: 84; 84/4160 = 2.0%] - newCrcFromRopeAux - crcFromRope - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 1/3391 Calls: 46/4160 = 1.1% [sum: 130; 130/4160 = 3.1%] - updateCrc32 - newCrcFromRopeAux - crcFromRope - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 2/3391 Calls: 41/4160 = 0.99% [sum: 171; 171/4160 = 4.1%] - updateCrc32 - updateCrc32 - newCrcFromRopeAux - crcFromRope - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 3/3391 Calls: 41/4160 = 0.99% [sum: 212; 212/4160 = 5.1%] - crcFromFile - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 4/3391 Calls: 41/4160 = 0.99% [sum: 253; 253/4160 = 6.1%] - updateCrc32 - crcFromFile - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 5/3391 Calls: 32/4160 = 0.77% [sum: 285; 285/4160 = 6.9%] - pop - newCrcFromRopeAux - crcFromRope - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - nim - Entry: 6/3391 Calls: 17/4160 = 0.41% [sum: 302; 302/4160 = 7.3%] - doOperation - forAllChildrenAux - pop - newCrcFromRopeAux - crcFromRope - writeRopeIfNotEqual - shouldRecompile - writeModule - myClose - closePasses - processModule - CompileModule - CompileProject - CommandCompileToC - MainCommand - HandleCmdLine - ... - nim - Entry: 7/3391 Calls: 14/4160 = 0.34% [sum: 316; 316/4160 = 7.6%] - Contains - isAccessible - interiorAllocatedPtr - gcMark - markStackAndRegisters - collectCTBody - collectCT - rawNewObj - newObj - newNode - copyTree - matchesAux - matches - resolveOverloads - semOverloadedCall - semOverloadedCallAnalyseEffects - ... - CommandCompileToC - MainCommand - HandleCmdLine diff --git a/doc/filelist.txt b/doc/filelist.txt index c863ba0cf..5522414fe 100644 --- a/doc/filelist.txt +++ b/doc/filelist.txt @@ -5,7 +5,7 @@ Short description of Nim's modules Module Description ============== ========================================================== nim main module: parses the command line and calls - ``main.MainCommand`` + `main.MainCommand` main implements the top-level command dispatching nimconf implements the config file reader syntaxes dispatcher for the different parsers and filters @@ -31,16 +31,14 @@ semstmts contains the semantic checking phase for statements semtypes contains the semantic checking phase for types seminst instantiation of generic procs and types semfold contains code to deal with constant folding -semthreads deep program analysis for threads -evals contains an AST interpreter for compile time evaluation +sempass2 Second semantic checking pass over the AST +vm contains an AST interpreter for compile time evaluation pragmas semantic checking of pragmas idents implements a general mapping from identifiers to an internal - representation (``PIdent``) that is used so that a simple + representation (`PIdent`) that is used so that a simple id-comparison suffices to establish whether two Nim identifiers are equivalent -ropes implements long strings represented as trees for - lazy evaluation; used mainly by the code generators transf transformations on the AST that need to be done before code generation diff --git a/doc/filters.md b/doc/filters.md new file mode 100644 index 000000000..9482b0b47 --- /dev/null +++ b/doc/filters.md @@ -0,0 +1,218 @@ +=================== +Source Code Filters +=================== + +.. include:: rstcommon.rst +.. default-role:: code +.. contents:: + +A `Source Code Filter (SCF)` transforms the input character stream to an in-memory +output stream before parsing. A filter can be used to provide templating +systems or preprocessors. + +To use a filter for a source file the `#?` notation is used: + + #? stdtmpl(subsChar = '$', metaChar = '#') + #proc generateXML(name, age: string): string = + # result = "" + <xml> + <name>$name</name> + <age>$age</age> + </xml> + +As the example shows, passing arguments to a filter can be done +just like an ordinary procedure call with named or positional arguments. The +available parameters depend on the invoked filter. Before version 0.12.0 of +the language `#!` was used instead of `#?`. + +**Hint:** With `--hint:codeBegin:on`:option: or `--verbosity:2`:option: +(or higher) while compiling or `nim check`:cmd:, Nim lists the processed code after +each filter application. + +Usage +===== + +First, put your SCF code in a separate file with filters specified in the first line. +**Note:** You can name your SCF file with any file extension you want, but the +conventional extension is `.nimf` +(it used to be `.tmpl` but that was too generic, for example preventing github to +recognize it as Nim source file). + +If we use `generateXML` code shown above and call the SCF file `xmlGen.nimf` +In your `main.nim`: + + ```nim + include "xmlGen.nimf" + + echo generateXML("John Smith","42") + ``` + +Pipe operator +============= + +Filters can be combined with the `|` pipe operator: + + #? strip(startswith="<") | stdtmpl + #proc generateXML(name, age: string): string = + # result = "" + <xml> + <name>$name</name> + <age>$age</age> + </xml> + + +Available filters +================= + +Replace filter +-------------- + +The replace filter replaces substrings in each line. + +Parameters and their defaults: + +* `sub: string = ""` + : the substring that is searched for + +* `by: string = ""` + : the string the substring is replaced with + + +Strip filter +------------ + +The strip filter simply removes leading and trailing whitespace from +each line. + +Parameters and their defaults: + +* `startswith: string = ""` + : strip only the lines that start with *startswith* (ignoring leading + whitespace). If empty every line is stripped. + +* `leading: bool = true` + : strip leading whitespace + +* `trailing: bool = true` + : strip trailing whitespace + + +StdTmpl filter +-------------- + +The stdtmpl filter provides a simple templating engine for Nim. The +filter uses a line based parser: Lines prefixed with a *meta character* +(default: `#`) contain Nim code, other lines are verbatim. Because +indentation-based parsing is not suited for a templating engine, control flow +statements need `end X` delimiters. + +Parameters and their defaults: + +* `metaChar: char = '#'` + : prefix for a line that contains Nim code + +* `subsChar: char = '$'` + : prefix for a Nim expression within a template line + +* `conc: string = " & "` + : the operation for concatenation + +* `emit: string = "result.add"` + : the operation to emit a string literal + +* `toString: string = "$"` + : the operation that is applied to each expression + +Example: + + #? stdtmpl | standard + #proc generateHTMLPage(title, currentTab, content: string, + # tabs: openArray[string]): string = + # result = "" + <head><title>$title</title></head> + <body> + <div id="menu"> + <ul> + #for tab in items(tabs): + #if currentTab == tab: + <li><a id="selected" + #else: + <li><a + #end if + href="${tab}.html">$tab</a></li> + #end for + </ul> + </div> + <div id="content"> + $content + A dollar: $$. + </div> + </body> + +The filter transforms this into: + + ```nim + proc generateHTMLPage(title, currentTab, content: string, + tabs: openArray[string]): string = + result = "" + result.add("<head><title>" & $(title) & "</title></head>\n" & + "<body>\n" & + " <div id=\"menu\">\n" & + " <ul>\n") + for tab in items(tabs): + if currentTab == tab: + result.add(" <li><a id=\"selected\" \n") + else: + result.add(" <li><a\n") + #end + result.add(" href=\"" & $(tab) & ".html\">" & $(tab) & "</a></li>\n") + #end + result.add(" </ul>\n" & + " </div>\n" & + " <div id=\"content\">\n" & + " " & $(content) & "\n" & + " A dollar: $.\n" & + " </div>\n" & + "</body>\n") + ``` + + +Each line that does not start with the meta character (ignoring leading +whitespace) is converted to a string literal that is added to `result`. + +The substitution character introduces a Nim expression *e* within the +string literal. *e* is converted to a string with the *toString* operation +which defaults to `$`. For strong type checking, set `toString` to the +empty string. *e* must match this PEG pattern: + + e <- [a-zA-Z\128-\255][a-zA-Z0-9\128-\255_.]* / '{' x '}' + x <- '{' x+ '}' / [^}]* + +To produce a single substitution character it has to be doubled: `$$` +produces `$`. + +The template engine is quite flexible. It is easy to produce a procedure that +writes the template code directly to a file: + + #? stdtmpl(emit="f.write") | standard + #proc writeHTMLPage(f: File, title, currentTab, content: string, + # tabs: openArray[string]) = + <head><title>$title</title></head> + <body> + <div id="menu"> + <ul> + #for tab in items(tabs): + #if currentTab == tab: + <li><a id="selected" + #else: + <li><a + #end if + href="${tab}.html" title = "$title - $tab">$tab</a></li> + #end for + </ul> + </div> + <div id="content"> + $content + A dollar: $$. + </div> + </body> diff --git a/doc/filters.rst b/doc/filters.rst deleted file mode 100644 index 40346ecaf..000000000 --- a/doc/filters.rst +++ /dev/null @@ -1,214 +0,0 @@ -=================== -Source Code Filters -=================== - -.. contents:: - -A `Source Code Filter (SCF)` transforms the input character stream to an in-memory -output stream before parsing. A filter can be used to provide templating -systems or preprocessors. - -To use a filter for a source file the ``#?`` notation is used:: - - #? stdtmpl(subsChar = '$', metaChar = '#') - #proc generateXML(name, age: string): string = - # result = "" - <xml> - <name>$name</name> - <age>$age</age> - </xml> - -As the example shows, passing arguments to a filter can be done -just like an ordinary procedure call with named or positional arguments. The -available parameters depend on the invoked filter. Before version 0.12.0 of -the language ``#!`` was used instead of ``#?``. - -**Hint:** With ``--hint[codeBegin]:on`` or ``--verbosity:2`` -(or higher) while compiling or `nim check`, Nim lists the processed code after -each filter application. - -Usage -===== - -First, put your SCF code in a separate file with filters specified in the first line. -**Note:** You can name your SCF file with any file extension you want, but the -conventional extension is ``.nimf`` -(it used to be ``.tmpl`` but that was too generic, for example preventing github to -recognize it as Nim source file). - -If we use `generateXML` code shown above and call the SCF file `xmlGen.nimf` -In your `main.nim`: - -.. code-block:: nim - include "xmlGen.nimf" - - echo generateXML("John Smith","42") - -Pipe operator -============= - -Filters can be combined with the ``|`` pipe operator:: - - #? strip(startswith="<") | stdtmpl - #proc generateXML(name, age: string): string = - # result = "" - <xml> - <name>$name</name> - <age>$age</age> - </xml> - - -Available filters -================= - -Replace filter --------------- - -The replace filter replaces substrings in each line. - -Parameters and their defaults: - - ``sub: string = ""`` - the substring that is searched for - - ``by: string = ""`` - the string the substring is replaced with - - -Strip filter ------------- - -The strip filter simply removes leading and trailing whitespace from -each line. - -Parameters and their defaults: - - ``startswith: string = ""`` - strip only the lines that start with *startswith* (ignoring leading - whitespace). If empty every line is stripped. - - ``leading: bool = true`` - strip leading whitespace - - ``trailing: bool = true`` - strip trailing whitespace - - -StdTmpl filter --------------- - -The stdtmpl filter provides a simple templating engine for Nim. The -filter uses a line based parser: Lines prefixed with a *meta character* -(default: ``#``) contain Nim code, other lines are verbatim. Because -indentation-based parsing is not suited for a templating engine, control flow -statements need ``end X`` delimiters. - -Parameters and their defaults: - - ``metaChar: char = '#'`` - prefix for a line that contains Nim code - - ``subsChar: char = '$'`` - prefix for a Nim expression within a template line - - ``conc: string = " & "`` - the operation for concatenation - - ``emit: string = "result.add"`` - the operation to emit a string literal - - ``toString: string = "$"`` - the operation that is applied to each expression - -Example:: - - #? stdtmpl | standard - #proc generateHTMLPage(title, currentTab, content: string, - # tabs: openArray[string]): string = - # result = "" - <head><title>$title</title></head> - <body> - <div id="menu"> - <ul> - #for tab in items(tabs): - #if currentTab == tab: - <li><a id="selected" - #else: - <li><a - #end if - href="${tab}.html">$tab</a></li> - #end for - </ul> - </div> - <div id="content"> - $content - A dollar: $$. - </div> - </body> - -The filter transforms this into: - -.. code-block:: nim - proc generateHTMLPage(title, currentTab, content: string, - tabs: openArray[string]): string = - result = "" - result.add("<head><title>" & $(title) & "</title></head>\n" & - "<body>\n" & - " <div id=\"menu\">\n" & - " <ul>\n") - for tab in items(tabs): - if currentTab == tab: - result.add(" <li><a id=\"selected\" \n") - else: - result.add(" <li><a\n") - #end - result.add(" href=\"" & $(tab) & ".html\">" & $(tab) & "</a></li>\n") - #end - result.add(" </ul>\n" & - " </div>\n" & - " <div id=\"content\">\n" & - " " & $(content) & "\n" & - " A dollar: $.\n" & - " </div>\n" & - "</body>\n") - - -Each line that does not start with the meta character (ignoring leading -whitespace) is converted to a string literal that is added to ``result``. - -The substitution character introduces a Nim expression *e* within the -string literal. *e* is converted to a string with the *toString* operation -which defaults to ``$``. For strong type checking, set ``toString`` to the -empty string. *e* must match this PEG pattern:: - - e <- [a-zA-Z\128-\255][a-zA-Z0-9\128-\255_.]* / '{' x '}' - x <- '{' x+ '}' / [^}]* - -To produce a single substitution character it has to be doubled: ``$$`` -produces ``$``. - -The template engine is quite flexible. It is easy to produce a procedure that -writes the template code directly to a file:: - - #? stdtmpl(emit="f.write") | standard - #proc writeHTMLPage(f: File, title, currentTab, content: string, - # tabs: openArray[string]) = - <head><title>$title</title></head> - <body> - <div id="menu"> - <ul> - #for tab in items(tabs): - #if currentTab == tab: - <li><a id="selected" - #else: - <li><a - #end if - href="${tab}.html" title = "$title - $tab">$tab</a></li> - #end for - </ul> - </div> - <div id="content"> - $content - A dollar: $$. - </div> - </body> diff --git a/doc/gc.rst b/doc/gc.rst deleted file mode 100644 index b118b1f8a..000000000 --- a/doc/gc.rst +++ /dev/null @@ -1,185 +0,0 @@ -======================= -Nim's Memory Management -======================= - -:Author: Andreas Rumpf -:Version: |nimversion| - -.. - - - "The road to hell is paved with good intentions." - - -Introduction -============ - -This document describes how the multi-paradigm memory management strategies work. -How to tune the garbage collectors for your needs, like (soft) `realtime systems`:idx:, -and how the memory management strategies that are not garbage collectors work. - - -Multi-paradigm Memory Management Strategies -=========================================== - -To choose the memory management strategy use the ``--gc:`` switch. - -- ``--gc:refc``. This is the default GC. It's a - deferred reference counting based garbage collector - with a simple Mark&Sweep backup GC in order to collect cycles. Heaps are thread local. -- ``--gc:markAndSweep``. Simple Mark-And-Sweep based garbage collector. Heaps are thread local. -- ``--gc:boehm``. Boehm based garbage collector, it offers a shared heap. -- ``--gc:go``. Go's garbage collector, useful for interoperability with Go. Offers a shared heap. -- ``--gc:arc``. Plain reference counting with - `move semantic optimizations <destructors.html#move-semantics>`_, offers a shared heap. - It offers deterministic performance for `hard realtime`:idx: systems. Reference cycles - cause memory leaks, beware. - -- ``--gc:orc``. Same as ``-gc:arc`` but adds a cycle collector based on "trial deletion". - Unfortunately that makes its performance profile hard to reason about so it is less - useful for hard realtime systems. - -- ``--gc:none``. No memory management strategy nor garbage collector. Allocated memory is - simply never freed. You should use ``--gc:arc`` instead. - - -================== ======== ================= ============== =================== -Memory Management Heap Reference Cycles Stop-The-World Command line switch -================== ======== ================= ============== =================== -RefC Local Cycle Collector No ``--gc:refc`` -Mark & Sweep Local Cycle Collector No ``--gc:markAndSweep`` -ARC Shared Leak No ``--gc:arc`` -ORC Shared Cycle Collector No ``--gc:orc`` -Boehm Shared Cycle Collector Yes ``--gc:boehm`` -Go Shared Cycle Collector Yes ``--gc:go`` -None Manual Manual Manual ``--gc:none`` -================== ======== ================= ============== =================== - -JavaScript's garbage collector is used for the `JavaScript and NodeJS -<backends.html#backends-the-javascript-target>`_ compilation targets. -The `NimScript <nims.html>`_ target uses the memory management strategy built into -the Nim compiler. - - -Tweaking the refc GC -==================== - -Cycle collector ---------------- - -The cycle collector can be en-/disabled independently from the other parts of -the garbage collector with ``GC_enableMarkAndSweep`` and ``GC_disableMarkAndSweep``. - - -Soft realtime support ---------------------- - -To enable realtime support, the symbol `useRealtimeGC`:idx: needs to be -defined via ``--define:useRealtimeGC`` (you can put this into your config -file as well). -With this switch the garbage collector supports the following operations: - -.. code-block:: nim - proc GC_setMaxPause*(maxPauseInUs: int) - proc GC_step*(us: int, strongAdvice = false, stackSize = -1) - -The unit of the parameters ``maxPauseInUs`` and ``us`` is microseconds. - -These two procs are the two modus operandi of the realtime garbage collector: - -(1) GC_SetMaxPause Mode - - You can call ``GC_SetMaxPause`` at program startup and then each triggered - garbage collector run tries to not take longer than ``maxPause`` time. However, it is - possible (and common) that the work is nevertheless not evenly distributed - as each call to ``new`` can trigger the garbage collector and thus take ``maxPause`` - time. - -(2) GC_step Mode - - This allows the garbage collector to perform some work for up to ``us`` time. - This is useful to call in a main loop to ensure the garbage collector can do its work. - To bind all garbage collector activity to a ``GC_step`` call, - deactivate the garbage collector with ``GC_disable`` at program startup. - If ``strongAdvice`` is set to ``true``, - then the garbage collector will be forced to perform collection cycle. - Otherwise, the garbage collector may decide not to do anything, - if there is not much garbage to collect. - You may also specify the current stack size via ``stackSize`` parameter. - It can improve performance, when you know that there are no unique Nim - references below certain point on the stack. Make sure the size you specify - is greater than the potential worst case size. - -These procs provide a "best effort" realtime guarantee; in particular the -cycle collector is not aware of deadlines. Deactivate it to get more -predictable realtime behaviour. Tests show that a 1ms max pause -time will be met in almost all cases on modern CPUs (with the cycle collector -disabled). - - -Time measurement with garbage collectors ----------------------------------------- - -The garbage collectors's way of measuring time uses -(see ``lib/system/timers.nim`` for the implementation): - -1) ``QueryPerformanceCounter`` and ``QueryPerformanceFrequency`` on Windows. -2) ``mach_absolute_time`` on Mac OS X. -3) ``gettimeofday`` on Posix systems. - -As such it supports a resolution of nanoseconds internally; however the API -uses microseconds for convenience. - -Define the symbol ``reportMissedDeadlines`` to make the -garbage collector output whenever it missed a deadline. -The reporting will be enhanced and supported by the API in later versions of the collector. - - -Tweaking the garbage collector ------------------------------- - -The collector checks whether there is still time left for its work after -every ``workPackage``'th iteration. This is currently set to 100 which means -that up to 100 objects are traversed and freed before it checks again. Thus -``workPackage`` affects the timing granularity and may need to be tweaked in -highly specialized environments or for older hardware. - - -Keeping track of memory -======================= - -If you need to pass around memory allocated by Nim to C, you can use the -procs ``GC_ref`` and ``GC_unref`` to mark objects as referenced to avoid them -being freed by the garbage collector. -Other useful procs from `system <system.html>`_ you can use to keep track of memory are: - -* ``getTotalMem()`` Returns the amount of total memory managed by the garbage collector. -* ``getOccupiedMem()`` Bytes reserved by the garbage collector and used by objects. -* ``getFreeMem()`` Bytes reserved by the garbage collector and not in use. -* ``GC_getStatistics()`` Garbage collector statistics as a human-readable string. - -These numbers are usually only for the running thread, not for the whole heap, -with the exception of ``--gc:boehm`` and ``--gc:go``. - -In addition to ``GC_ref`` and ``GC_unref`` you can avoid the garbage collector by manually -allocating memory with procs like ``alloc``, ``alloc0``, ``allocShared``, ``allocShared0`` or ``allocCStringArray``. -The garbage collector won't try to free them, you need to call their respective *dealloc* pairs -(``dealloc``, ``deallocShared``, ``deallocCStringArray``, etc) -when you are done with them or they will leak. - - -Heap dump -========= - -The heap dump feature is still in its infancy, but it already proved -useful for us, so it might be useful for you. To get a heap dump, compile -with ``-d:nimTypeNames`` and call ``dumpNumberOfInstances`` at a strategic place in your program. -This produces a list of used types in your program and for every type -the total amount of object instances for this type as well as the total -amount of bytes these instances take up. This list is currently unsorted! -You need to use external shell script hacking to sort it. - -The numbers count the number of objects in all garbage collector heaps, they refer to -all running threads, not only to the current thread. (The current thread -would be the thread that calls ``dumpNumberOfInstances``.) This might -change in later versions. diff --git a/doc/grammar.txt b/doc/grammar.txt index 9d952d372..51b3e0053 100644 --- a/doc/grammar.txt +++ b/doc/grammar.txt @@ -1,13 +1,13 @@ # This file is generated by compiler/parser.nim. -module = stmt ^* (';' / IND{=}) +module = complexOrSimpleStmt ^* (';' / IND{=}) comma = ',' COMMENT? semicolon = ';' COMMENT? colon = ':' COMMENT? colcom = ':' COMMENT? operator = OP0 | OP1 | OP2 | OP3 | OP4 | OP5 | OP6 | OP7 | OP8 | OP9 | 'or' | 'xor' | 'and' - | 'is' | 'isnot' | 'in' | 'notin' | 'of' | 'as' | 'from' | - | 'div' | 'mod' | 'shl' | 'shr' | 'not' | 'static' | '..' + | 'is' | 'isnot' | 'in' | 'notin' | 'of' | 'as' | 'from' + | 'div' | 'mod' | 'shl' | 'shr' | 'not' | '..' prefixOperator = operator optInd = COMMENT? IND? optPar = (IND{>} | IND{=})? @@ -22,69 +22,70 @@ ampExpr = plusExpr (OP7 optInd plusExpr)* plusExpr = mulExpr (OP8 optInd mulExpr)* mulExpr = dollarExpr (OP9 optInd dollarExpr)* dollarExpr = primary (OP10 optInd primary)* +operatorB = OP0 | OP1 | OP2 | OP3 | OP4 | OP5 | OP6 | OP7 | OP8 | OP9 | + 'div' | 'mod' | 'shl' | 'shr' | 'in' | 'notin' | + 'is' | 'isnot' | 'not' | 'of' | 'as' | 'from' | '..' | 'and' | 'or' | 'xor' symbol = '`' (KEYW|IDENT|literal|(operator|'('|')'|'['|']'|'{'|'}'|'=')+)+ '`' - | IDENT | KEYW -exprColonEqExpr = expr (':'|'=' expr)? + | IDENT | 'addr' | 'type' | 'static' +symbolOrKeyword = symbol | KEYW +exprColonEqExpr = expr ((':'|'=') expr + / doBlock extraPostExprBlock*)? +exprEqExpr = expr ('=' expr + / doBlock extraPostExprBlock*)? exprList = expr ^+ comma +optionalExprList = expr ^* comma exprColonEqExprList = exprColonEqExpr (comma exprColonEqExpr)* (comma)? -dotExpr = expr '.' optInd (symbol | '[:' exprList ']') -explicitGenericInstantiation = '[:' exprList ']' ( '(' exprColonEqExpr ')' )? -qualifiedIdent = symbol ('.' optInd symbol)? +qualifiedIdent = symbol ('.' optInd symbolOrKeyword)? setOrTableConstr = '{' ((exprColonEqExpr comma)* | ':' ) '}' -castExpr = 'cast' '[' optInd typeDesc optPar ']' '(' optInd expr optPar ')' +castExpr = 'cast' ('[' optInd typeDesc optPar ']' '(' optInd expr optPar ')') / parKeyw = 'discard' | 'include' | 'if' | 'while' | 'case' | 'try' | 'finally' | 'except' | 'for' | 'block' | 'const' | 'let' | 'when' | 'var' | 'mixin' par = '(' optInd - ( &parKeyw complexOrSimpleStmt ^+ ';' - | ';' complexOrSimpleStmt ^+ ';' + ( &parKeyw (ifExpr / complexOrSimpleStmt) ^+ ';' + | ';' (ifExpr / complexOrSimpleStmt) ^+ ';' | pragmaStmt - | simpleExpr ( ('=' expr (';' complexOrSimpleStmt ^+ ';' )? ) + | simpleExpr ( (doBlock extraPostExprBlock*) + | ('=' expr (';' (ifExpr / complexOrSimpleStmt) ^+ ';' )? ) | (':' expr (',' exprColonEqExpr ^+ ',' )? ) ) ) optPar ')' literal = | INT_LIT | INT8_LIT | INT16_LIT | INT32_LIT | INT64_LIT | UINT_LIT | UINT8_LIT | UINT16_LIT | UINT32_LIT | UINT64_LIT | FLOAT_LIT | FLOAT32_LIT | FLOAT64_LIT | STR_LIT | RSTR_LIT | TRIPLESTR_LIT - | CHAR_LIT + | CHAR_LIT | CUSTOM_NUMERIC_LIT | NIL generalizedLit = GENERALIZED_STR_LIT | GENERALIZED_TRIPLESTR_LIT identOrLiteral = generalizedLit | symbol | literal - | par | arrayConstr | setOrTableConstr + | par | arrayConstr | setOrTableConstr | tupleConstr | castExpr tupleConstr = '(' optInd (exprColonEqExpr comma?)* optPar ')' arrayConstr = '[' optInd (exprColonEqExpr comma?)* optPar ']' primarySuffix = '(' (exprColonEqExpr comma?)* ')' - | '.' optInd symbol generalizedLit? + | '.' optInd symbolOrKeyword ('[:' exprList ']' ( '(' exprColonEqExpr ')' )?)? generalizedLit? + | DOTLIKEOP optInd symbolOrKeyword generalizedLit? | '[' optInd exprColonEqExprList optPar ']' | '{' optInd exprColonEqExprList optPar '}' - | &( '`'|IDENT|literal|'cast'|'addr'|'type') expr # command syntax -condExpr = expr colcom expr optInd - ('elif' expr colcom expr optInd)* - 'else' colcom expr -ifExpr = 'if' condExpr -whenExpr = 'when' condExpr pragma = '{.' optInd (exprColonEqExpr comma?)* optPar ('.}' | '}') identVis = symbol OPR? # postfix position -identVisDot = symbol '.' optInd symbol OPR? +identVisDot = symbol '.' optInd symbolOrKeyword OPR? identWithPragma = identVis pragma? identWithPragmaDot = identVisDot pragma? declColonEquals = identWithPragma (comma identWithPragma)* comma? - (':' optInd typeDesc)? ('=' optInd expr)? + (':' optInd typeDescExpr)? ('=' optInd expr)? identColonEquals = IDENT (comma IDENT)* comma? - (':' optInd typeDesc)? ('=' optInd expr)?) -inlTupleDecl = 'tuple' - '[' optInd (identColonEquals (comma/semicolon)?)* optPar ']' -extTupleDecl = 'tuple' - COMMENT? (IND{>} identColonEquals (IND{=} identColonEquals)*)? -tupleClass = 'tuple' + (':' optInd typeDescExpr)? ('=' optInd expr)?) +tupleTypeBracket = '[' optInd (identColonEquals (comma/semicolon)?)* optPar ']' +tupleType = 'tuple' tupleTypeBracket +tupleDecl = 'tuple' (tupleTypeBracket / + COMMENT? (IND{>} identColonEquals (IND{=} identColonEquals)*)?) paramList = '(' declColonEquals ^* (comma/semicolon) ')' paramListArrow = paramList? ('->' optInd typeDesc)? paramListColon = paramList? (':' optInd typeDesc)? doBlock = 'do' paramListArrow pragma? colcom stmt -procExpr = 'proc' paramListColon pragma? ('=' COMMENT? stmt)? -distinct = 'distinct' optInd typeDesc -forStmt = 'for' (identWithPragma ^+ comma) 'in' expr colcom stmt +routineExpr = ('proc' | 'func' | 'iterator') paramListColon pragma? ('=' COMMENT? stmt)? +routineType = ('proc' | 'iterator') paramListColon pragma? +forStmt = 'for' ((varTuple / identWithPragma) ^+ comma) 'in' expr colcom stmt forExpr = forStmt expr = (blockExpr | ifExpr @@ -93,24 +94,33 @@ expr = (blockExpr | forExpr | tryExpr) / simpleExpr -typeKeyw = 'var' | 'out' | 'ref' | 'ptr' | 'shared' | 'tuple' - | 'proc' | 'iterator' | 'distinct' | 'object' | 'enum' -primary = typeKeyw optInd typeDesc - / prefixOperator* identOrLiteral primarySuffix* - / 'bind' primary -typeDesc = simpleExpr ('not' expr)? -typeDefAux = simpleExpr ('not' expr)? - | 'concept' typeClass -postExprBlocks = ':' stmt? ( IND{=} doBlock - | IND{=} 'of' exprList ':' stmt - | IND{=} 'elif' expr ':' stmt - | IND{=} 'except' exprList ':' stmt - | IND{=} 'else' ':' stmt )* -exprStmt = simpleExpr - (( '=' optInd expr colonBody? ) - / ( expr ^+ comma - postExprBlocks - ))? +simplePrimary = SIGILLIKEOP? identOrLiteral primarySuffix* +commandStart = &('`'|IDENT|literal|'cast'|'addr'|'type'|'var'|'out'| + 'static'|'enum'|'tuple'|'object'|'proc') +primary = simplePrimary (commandStart expr (doBlock extraPostExprBlock*)?)? + / operatorB primary + / routineExpr + / rawTypeDesc + / prefixOperator primary +rawTypeDesc = (tupleType | routineType | 'enum' | 'object' | + ('var' | 'out' | 'ref' | 'ptr' | 'distinct') typeDesc?) + ('not' primary)? +typeDescExpr = (routineType / simpleExpr) ('not' primary)? +typeDesc = rawTypeDesc / typeDescExpr +typeDefValue = ((tupleDecl | enumDecl | objectDecl | conceptDecl | + ('ref' | 'ptr' | 'distinct') (tupleDecl | objectDecl)) + / (simpleExpr (exprEqExpr ^+ comma postExprBlocks?)?)) + ('not' primary)? +extraPostExprBlock = ( IND{=} doBlock + | IND{=} 'of' exprList ':' stmt + | IND{=} 'elif' expr ':' stmt + | IND{=} 'except' optionalExprList ':' stmt + | IND{=} 'finally' ':' stmt + | IND{=} 'else' ':' stmt ) +postExprBlocks = (doBlock / ':' (extraPostExprBlock / stmt)) extraPostExprBlock* +exprStmt = simpleExpr postExprBlocks? + / simplePrimary (exprEqExpr ^+ comma) postExprBlocks? + / simpleExpr '=' optInd (expr postExprBlocks?) importStmt = 'import' optInd expr ((comma expr)* / 'except' optInd (expr ^+ comma)) @@ -124,12 +134,17 @@ raiseStmt = 'raise' optInd expr? yieldStmt = 'yield' optInd expr? discardStmt = 'discard' optInd expr? breakStmt = 'break' optInd expr? -continueStmt = 'break' optInd expr? +continueStmt = 'continue' optInd expr? condStmt = expr colcom stmt COMMENT? (IND{=} 'elif' expr colcom stmt)* (IND{=} 'else' colcom stmt)? ifStmt = 'if' condStmt whenStmt = 'when' condStmt +condExpr = expr colcom stmt optInd + ('elif' expr colcom stmt optInd)* + 'else' colcom stmt +ifExpr = 'if' condExpr +whenExpr = 'when' condExpr whileStmt = 'while' expr colcom stmt ofBranch = 'of' exprList colcom stmt ofBranches = ofBranch (IND{=} ofBranch)* @@ -139,12 +154,11 @@ caseStmt = 'case' expr ':'? COMMENT? (IND{>} ofBranches DED | IND{=} ofBranches) tryStmt = 'try' colcom stmt &(IND{=}? 'except'|'finally') - (IND{=}? 'except' exprList colcom stmt)* + (IND{=}? 'except' optionalExprList colcom stmt)* (IND{=}? 'finally' colcom stmt)? tryExpr = 'try' colcom stmt &(optInd 'except'|'finally') - (optInd 'except' exprList colcom stmt)* + (optInd 'except' optionalExprList colcom stmt)* (optInd 'finally' colcom stmt)? -exceptBlock = 'except' colcom stmt blockStmt = 'block' symbol? colcom stmt blockExpr = 'block' symbol? colcom stmt staticStmt = 'static' colcom stmt @@ -159,7 +173,7 @@ routine = optInd identVis pattern? genericParamList? paramListColon pragma? ('=' COMMENT? stmt)? indAndComment commentStmt = COMMENT section(RULE) = COMMENT? RULE / (IND{>} (RULE / COMMENT)^+IND{=} DED) -enum = 'enum' optInd (symbol pragma? optInd ('=' optInd expr COMMENT?)? comma?)+ +enumDecl = 'enum' optInd (symbol pragma? optInd ('=' optInd expr COMMENT?)? comma?)+ objectWhen = 'when' expr colcom objectPart COMMENT? ('elif' expr colcom objectPart COMMENT?)* ('else' colcom objectPart COMMENT?)? @@ -167,19 +181,19 @@ objectBranch = 'of' exprList colcom objectPart objectBranches = objectBranch (IND{=} objectBranch)* (IND{=} 'elif' expr colcom objectPart)* (IND{=} 'else' colcom objectPart)? -objectCase = 'case' identWithPragma ':' typeDesc ':'? COMMENT? +objectCase = 'case' declColonEquals ':'? COMMENT? (IND{>} objectBranches DED | IND{=} objectBranches) objectPart = IND{>} objectPart^+IND{=} DED / objectWhen / objectCase / 'nil' / 'discard' / declColonEquals -object = 'object' pragma? ('of' typeDesc)? COMMENT? objectPart -typeClassParam = ('var' | 'out')? symbol -typeClass = typeClassParam ^* ',' (pragma)? ('of' typeDesc ^* ',')? +objectDecl = 'object' ('of' typeDesc)? COMMENT? objectPart +conceptParam = ('var' | 'out' | 'ptr' | 'ref' | 'static' | 'type')? symbol +conceptDecl = 'concept' conceptParam ^* ',' (pragma)? ('of' typeDesc ^* ',')? &IND{>} stmt -typeDef = identWithPragmaDot genericParamList? '=' optInd typeDefAux - indAndComment? / identVisDot genericParamList? pragma '=' optInd typeDefAux +typeDef = identVisDot genericParamList? pragma '=' optInd typeDefValue indAndComment? -varTuple = '(' optInd identWithPragma ^+ comma optPar ')' '=' optInd expr +varTupleLhs = '(' optInd (identWithPragma / varTupleLhs) ^+ comma optPar ')' (':' optInd typeDescExpr)? +varTuple = varTupleLhs '=' optInd expr colonBody = colcom stmt postExprBlocks? variable = (varTuple / identColonEquals) colonBody? indAndComment constant = (varTuple / identWithPragma) (colon typeDesc)? '=' optInd expr indAndComment @@ -193,8 +207,8 @@ complexOrSimpleStmt = (ifStmt | whenStmt | whileStmt | tryStmt | forStmt | blockStmt | staticStmt | deferStmt | asmStmt | 'proc' routine - | 'func' routine | 'method' routine + | 'func' routine | 'iterator' routine | 'macro' routine | 'template' routine diff --git a/doc/hcr.rst b/doc/hcr.md index 567d77178..285a86282 100644 --- a/doc/hcr.rst +++ b/doc/hcr.md @@ -2,6 +2,9 @@ Hot code reloading =================================== +.. default-role:: code +.. include:: rstcommon.rst + The `hotCodeReloading`:idx: option enables special compilation mode where changes in the code can be applied automatically to a running program. The code reloading happens at the granularity of an individual module. @@ -14,22 +17,21 @@ preserved. Basic workflow ============== -Currently hot code reloading does not work for the main module itself, +Currently, hot code reloading does not work for the main module itself, so we have to use a helper module where the major logic we want to change during development resides. -In this example we use SDL2 to create a window and we reload the logic -code when ``F9`` is pressed. The important lines are marked with ``#***``. -To install SDL2 you can use ``nimble install sdl2``. - +In this example, we use SDL2 to create a window and we reload the logic +code when `F9` is pressed. The important lines are marked with `#***`. +To install SDL2 you can use `nimble install sdl2`:cmd:. -.. code-block:: nim + ```nim # logic.nim import sdl2 #*** import the hotcodereloading stdlib module *** - import hotcodereloading + import std/hotcodereloading var runGame*: bool = true var window: WindowPtr @@ -76,14 +78,14 @@ To install SDL2 you can use ``nimble install sdl2``. discard renderer.clear() discard renderer.setDrawColor(255, 128, 128, 0) - var rect = Rect(x: posX - 25, y: posY - 25, w: 50.cint, h: 50.cint) + var rect: Rect = (x: posX - 25, y: posY - 25, w: 50.cint, h: 50.cint) discard renderer.fillRect(rect) delay(16) renderer.present() + ``` -.. code-block:: nim - + ```nim # mymain.nim import logic @@ -94,38 +96,46 @@ To install SDL2 you can use ``nimble install sdl2``. destroy() main() + ``` -Compile this example via:: +Compile this example via: + ```cmd nim c --hotcodereloading:on mymain.nim + ``` Now start the program and KEEP it running! -:: - + ```cmd # Unix: mymain & # or Windows (click on the .exe) mymain.exe # edit + ``` -For example, change the line:: +For example, change the line: + ```nim discard renderer.setDrawColor(255, 128, 128, 0) + ``` -into:: +into: + ```nim discard renderer.setDrawColor(255, 255, 128, 0) + ``` (This will change the color of the rectangle.) Then recompile the project, but do not restart or quit the mymain.exe program! -:: + ```cmd nim c --hotcodereloading:on mymain.nim + ``` -Now give the ``mymain`` SDL window the focus, press F9 and watch the +Now give the `mymain` SDL window the focus, press F9, and watch the updated version of the program. @@ -133,11 +143,11 @@ updated version of the program. Reloading API ============= -One can use the special event handlers ``beforeCodeReload`` and -``afterCodeReload`` to reset the state of a particular variable or to force +One can use the special event handlers `beforeCodeReload` and +`afterCodeReload` to reset the state of a particular variable or to force the execution of certain statements: -.. code-block:: Nim + ```Nim var settings = initTable[string, string]() lastReload: Time @@ -150,6 +160,7 @@ the execution of certain statements: afterCodeReload: lastReload = now() resetProgramState() + ``` On each code reload, Nim will first execute all `beforeCodeReload`:idx: handlers registered in the previous version of the program and then all @@ -158,7 +169,7 @@ that any handlers appearing in modules that weren't reloaded will also be executed. To prevent this behavior, one can guard the code with the `hasModuleChanged()`:idx: API: -.. code-block:: Nim + ```Nim import mydb var myCache = initTable[Key, Value]() @@ -166,6 +177,7 @@ executed. To prevent this behavior, one can guard the code with the afterCodeReload: if hasModuleChanged(mydb): resetCache(myCache) + ``` The hot code reloading is based on dynamic library hot swapping in the native targets and direct manipulation of the global namespace in the JavaScript @@ -178,24 +190,23 @@ It's expected that most projects will implement the reloading with a suitable build-system triggered IPC notification mechanism, but a polling solution is also possible through the provided `hasAnyModuleChanged()`:idx: API. -In order to access ``beforeCodeReload``, ``afterCodeReload``, ``hasModuleChanged`` -or ``hasAnyModuleChanged`` one must import the `hotcodereloading`:idx: module. +In order to access `beforeCodeReload`, `afterCodeReload`, `hasModuleChanged` +or `hasAnyModuleChanged` one must import the `hotcodereloading`:idx: module. Native code targets =================== Native projects using the hot code reloading option will be implicitly -compiled with the `-d:useNimRtl` option and they will depend on both -the ``nimrtl`` library and the ``nimhcr`` library which implements the -hot code reloading run-time. Both libraries can be found in the ``lib`` +compiled with the `-d:useNimRtl`:option: option and they will depend on both +the `nimrtl` library and the `nimhcr` library which implements the +hot code reloading run-time. Both libraries can be found in the `lib` folder of Nim and can be compiled into dynamic libraries to satisfy -runtime demands of the example code above. An example of compiling +runtime demands of the example code above. An example of compiling ``nimhcr.nim`` and ``nimrtl.nim`` when the source dir of Nim is installed with choosenim follows. -:: - + ```console # Unix/MacOS # Make sure you are in the directory containing your .nim files $ cd your-source-directory @@ -206,18 +217,19 @@ with choosenim follows. # verify that you have two files named libnimhcr and libnimrtl in your # source directory (.dll for Windows, .so for Unix, .dylib for MacOS) + ``` All modules of the project will be compiled to separate dynamic link -libraries placed in the ``nimcache`` directory. Please note that during +libraries placed in the `nimcache` directory. Please note that during the execution of the program, the hot code reloading run-time will load only copies of these libraries in order to not interfere with any newly issued build commands. The main module of the program is considered non-reloadable. Please note that procs from reloadable modules should not appear in the call stack of -program while ``performCodeReload`` is being called. Thus, the main module +program while `performCodeReload` is being called. Thus, the main module is a suitable place for implementing a program loop capable of calling -``performCodeReload``. +`performCodeReload`. Please note that reloading won't be possible when any of the type definitions in the program has been changed. When closure iterators are used (directly or @@ -228,7 +240,6 @@ completion. JavaScript target ================= -Once your code is compiled for hot reloading, the ``nim-livereload`` NPM -package provides a convenient solution for implementing the actual reloading +Once your code is compiled for hot reloading, a convenient solution for implementing the actual reloading in the browser using a framework such as [LiveReload](http://livereload.com/) or [BrowserSync](https://browsersync.io/). diff --git a/doc/idetools.rst b/doc/idetools.md index ac3b92947..0388a76c0 100644 --- a/doc/idetools.rst +++ b/doc/idetools.md @@ -1,3 +1,5 @@ +.. default-role:: code + ================================ Nim IDE Integration Guide ================================ @@ -8,24 +10,21 @@ .. contents:: -.. raw:: html - <blockquote><p> - "yes, I'm the creator" -- Araq, 2013-07-26 19:28:32. - </p></blockquote> +> "yes, I'm the creator" -- Araq, 2013-07-26 19:28:32. -Note: this is mostly outdated, see instead `nimsuggest <nimsuggest.html>`_ +Note: this is mostly outdated, see instead [nimsuggest](nimsuggest.html) Nim differs from many other compilers in that it is really fast, and being so fast makes it suited to provide external queries for text editors about the source code being written. Through the -``idetools`` command of `the compiler <nimc.html>`_, any IDE -can query a ``.nim`` source file and obtain useful information like +`idetools` command of [the compiler](nimc.html), any IDE +can query a `.nim` source file and obtain useful information like definition of symbols or suggestions for completion. This document will guide you through the available options. If you want to look at practical examples of idetools support you can look -at the test files found in the `Test suite`_ or `various editor -integrations <https://github.com/Araq/Nim/wiki/Editor-Support>`_ +at the test files found in the [Test suite] or [various editor +integrations](https://github.com/Araq/Nim/wiki/Editor-Support) already available. @@ -36,44 +35,48 @@ Specifying the location of the query ------------------------------------ All of the available idetools commands require you to specify a -query location through the ``--track`` or ``--trackDirty`` switches. -The general idetools invocations are:: +query location through the `--track` or `--trackDirty` switches. +The general idetools invocations are: - nim idetools --track:FILE,LINE,COL <switches> proj.nim + ```cmd + nim idetools --track:FILE,LINE,COL <switches> proj.nim + ``` -Or:: +Or: - nim idetools --trackDirty:DIRTY_FILE,FILE,LINE,COL <switches> proj.nim + ```cmd + nim idetools --trackDirty:DIRTY_FILE,FILE,LINE,COL <switches> proj.nim + ``` -``proj.nim`` - This is the main *project* filename. Most of the time you will +`proj.nim` +: This is the main *project* filename. Most of the time you will pass in the same as **FILE**, but for bigger projects this is the file which is used as main entry point for the program, the one which users compile to generate a final binary. -``<switches>`` - This would be any of the other idetools available options, like - ``--def`` or ``--suggest`` explained in the following sections. +`<switches>` +: This would be any of the other idetools available options, like + `--def` or `--suggest` explained in the following sections. -``COL`` - An integer with the column you are going to query. For the +`COL` +: An integer with the column you are going to query. For the compiler columns start at zero, so the first column will be **0** and the last in an 80 column terminal will be **79**. -``LINE`` - An integer with the line you are going to query. For the compiler +`LINE` +: An integer with the line you are going to query. For the compiler lines start at **1**. -``FILE`` - The file you want to perform the query on. Usually you will +`FILE` +: The file you want to perform the query on. Usually you will pass in the same value as **proj.nim**. -``DIRTY_FILE`` - The **FILE** parameter is enough for static analysis, but IDEs +`DIRTY_FILE` +: The **FILE** parameter is enough for static analysis, but IDEs tend to have *unsaved buffers* where the user may still be in the middle of typing a line. In such situations the IDE can save the current contents to a temporary file and then use the - ``--trackDirty`` switch. + `--trackDirty` switch. Dirty files are likely to contain errors and they are usually compiled partially only to the point needed to service the @@ -91,7 +94,7 @@ Or:: Definitions ----------- -The ``--def`` idetools switch performs a query about the definition +The `--def` idetools switch performs a query about the definition of a specific symbol. If available, idetools will answer with the type, source file, line/column information and other accessory data if available like a docstring. With this information an IDE can @@ -112,7 +115,7 @@ can't find any valid symbol matching the position of the query. Suggestions ----------- -The ``--suggest`` idetools switch performs a query about possible +The `--suggest` idetools switch performs a query about possible completion symbols at some point in the file. IDEs can easily provide an autocompletion feature where the IDE scans the current file (and related ones, if it knows about the language being edited and follows @@ -125,8 +128,8 @@ to the case insensitiveness of the language (plus underscores as separators!). The typical usage scenario for this option is to call it after the -user has typed the dot character for `the object oriented call -syntax <tut2.html#object-oriented-programming-method-call-syntax>`_. +user has typed the dot character for [the object oriented call +syntax](tut2.html#object-oriented-programming-method-call-syntax). Idetools will try to return the suggestions sorted first by scope (from innermost to outermost) and then by item name. @@ -134,7 +137,7 @@ Idetools will try to return the suggestions sorted first by scope Invocation context ------------------ -The ``--context`` idetools switch is very similar to the suggestions +The `--context` idetools switch is very similar to the suggestions switch, but instead of being used after the user has typed a dot character, this one is meant to be used after the user has typed an opening brace to start typing parameters. @@ -143,7 +146,7 @@ an opening brace to start typing parameters. Symbol usages ------------- -The ``--usages`` idetools switch lists all usages of the symbol at +The `--usages` idetools switch lists all usages of the symbol at a position. IDEs can use this to find all the places in the file where the symbol is used and offer the user to rename it in all places at the same time. Again, a pure string based search and @@ -179,14 +182,18 @@ results of the compilation, and subsequent queries should be fast in the millisecond range, thus being responsive enough for IDEs. If you want to start the server using stdin/stdout as communication -you need to type:: +you need to type: - nim serve --server.type:stdin proj.nim + ```cmd + nim serve --server.type:stdin proj.nim + ``` -If you want to start the server using tcp and a port, you need to type:: +If you want to start the server using tcp and a port, you need to type: - nim serve --server.type:tcp --server.port:6000 \ + ```cmd + nim serve --server.type:tcp --server.port:6000 \ --server.address:hostname proj.nim + ``` In both cases the server will start up and await further commands. The syntax of the commands you can now send to the server is @@ -197,7 +204,7 @@ of text it thinks necessary plus an empty line to indicate the end of the answer. You can find examples of client/server communication in the idetools -tests found in the `Test suite`_. +tests found in the [Test suite]. Parsing idetools output @@ -207,15 +214,15 @@ Idetools outputs is always returned on single lines separated by tab characters (``\t``). The values of each column are: 1. Three characters indicating the type of returned answer (e.g. - def for definition, ``sug`` for suggestion, etc). -2. Type of the symbol. This can be ``skProc``, ``skLet``, and just - about any of the enums defined in the module ``compiler/ast.nim``. + def for definition, `sug` for suggestion, etc). +2. Type of the symbol. This can be `skProc`, `skLet`, and just + about any of the enums defined in the module `compiler/ast.nim`. 3. Full qualified path of the symbol. If you are querying a symbol - defined in the ``proj.nim`` file, this would have the form - ``proj.symbolName``. + defined in the `proj.nim` file, this would have the form + `proj.symbolName`. 4. Type/signature. For variables and enums this will contain the type of the symbol, for procs, methods and templates this will - contain the full unique signature (e.g. ``proc (File)``). + contain the full unique signature (e.g. `proc (File)`). 5. Full path to the file containing the symbol. 6. Line where the symbol is located in the file. Lines start to count at **1**. @@ -240,45 +247,48 @@ symbol for which idetools returns valid output. skConst ------- -| **Third column**: module + [n scope nesting] + const name. +| **Third column**: module + \[n scope nesting] + const name. | **Fourth column**: the type of the const value. | **Docstring**: always the empty string. -.. code-block:: nim - const SOME_SEQUENCE = @[1, 2] - --> col 2: $MODULE.SOME_SEQUENCE - col 3: seq[int] - col 7: "" + ```nim + const SOME_SEQUENCE = @[1, 2] + --> col 2: $MODULE.SOME_SEQUENCE + col 3: seq[int] + col 7: "" + ``` skEnumField ----------- -| **Third column**: module + [n scope nesting] + enum type + enum field name. +| **Third column**: module + \[n scope nesting] + enum type + enum field name. | **Fourth column**: enum type grouping other enum fields. | **Docstring**: always the empty string. -.. code-block:: nim - Open(filename, fmWrite) - --> col 2: system.FileMode.fmWrite - col 3: FileMode - col 7: "" + ```nim + Open(filename, fmWrite) + --> col 2: system.FileMode.fmWrite + col 3: FileMode + col 7: "" + ``` skForVar -------- -| **Third column**: module + [n scope nesting] + var name. +| **Third column**: module + \[n scope nesting] + var name. | **Fourth column**: type of the var. | **Docstring**: always the empty string. -.. code-block:: nim - proc looper(filename = "tests.nim") = - for letter in filename: - echo letter - --> col 2: $MODULE.looper.letter - col 3: char - col 7: "" + ```nim + proc looper(filename = "tests.nim") = + for letter in filename: + echo letter + --> col 2: $MODULE.looper.letter + col 3: char + col 7: "" + ``` skIterator, skClosureIterator @@ -289,48 +299,51 @@ defined, since at that point in the file the parser hasn't processed the full line yet. The signature will be returned complete in posterior instances of the iterator. -| **Third column**: module + [n scope nesting] + iterator name. +| **Third column**: module + \[n scope nesting] + iterator name. | **Fourth column**: signature of the iterator including return type. | **Docstring**: docstring if available. -.. code-block:: nim - let - text = "some text" - letters = toSeq(runes(text)) - --> col 2: unicode.runes - col 3: iterator (string): Rune - col 7: "iterates over any unicode character of the string `s`." + ```nim + let + text = "some text" + letters = toSeq(runes(text)) + --> col 2: unicode.runes + col 3: iterator (string): Rune + col 7: "iterates over any unicode character of the string `s`." + ``` skLabel ------- -| **Third column**: module + [n scope nesting] + name. +| **Third column**: module + \[n scope nesting] + name. | **Fourth column**: always the empty string. | **Docstring**: always the empty string. -.. code-block:: nim - proc test(text: string) = - var found = -1 - block loops: - --> col 2: $MODULE.test.loops - col 3: "" - col 7: "" + ```nim + proc test(text: string) = + var found = -1 + block loops: + --> col 2: $MODULE.test.loops + col 3: "" + col 7: "" + ``` skLet ----- -| **Third column**: module + [n scope nesting] + let name. +| **Third column**: module + \[n scope nesting] + let name. | **Fourth column**: the type of the let variable. | **Docstring**: always the empty string. -.. code-block:: nim - let - text = "some text" - --> col 2: $MODULE.text - col 3: TaintedString - col 7: "" + ```nim + let + text = "some text" + --> col 2: $MODULE.text + col 3: string + col 7: "" + ``` skMacro @@ -341,16 +354,17 @@ defined, since at that point in the file the parser hasn't processed the full line yet. The signature will be returned complete in posterior instances of the macro. -| **Third column**: module + [n scope nesting] + macro name. +| **Third column**: module + \[n scope nesting] + macro name. | **Fourth column**: signature of the macro including return type. | **Docstring**: docstring if available. -.. code-block:: nim - proc testMacro() = - expect(EArithmetic): - --> col 2: idetools_api.expect - col 3: proc (varargs[expr], stmt): stmt - col 7: "" + ```nim + proc testMacro() = + expect(EArithmetic): + --> col 2: idetools_api.expect + col 3: proc (varargs[expr], stmt): stmt + col 7: "" + ``` skMethod @@ -361,8 +375,8 @@ defined, since at that point in the file the parser hasn't processed the full line yet. The signature will be returned complete in posterior instances of the method. -Methods imply `dynamic dispatch -<tut2.html#object-oriented-programming-dynamic-dispatch>`_ and +Methods imply [dynamic dispatch]( +tut2.html#object-oriented-programming-dynamic-dispatch) and idetools performs a static analysis on the code. For this reason idetools may not return the definition of the correct method you are querying because it may be impossible to know until the code @@ -374,37 +388,39 @@ While at the language level a method is differentiated from others by the parameters and return value, the signature of the method returned by idetools returns also the pragmas for the method. -Note that at the moment the word ``proc`` is returned for the -signature of the found method instead of the expected ``method``. +Note that at the moment the word `proc` is returned for the +signature of the found method instead of the expected `method`. This may change in the future. -| **Third column**: module + [n scope nesting] + method name. +| **Third column**: module + \[n scope nesting] + method name. | **Fourth column**: signature of the method including return type. | **Docstring**: docstring if available. -.. code-block:: nim - method eval(e: PExpr): int = quit "to override!" - method eval(e: PLiteral): int = e.x - method eval(e: PPlusExpr): int = eval(e.a) + eval(e.b) - echo eval(newPlus(newPlus(newLit(1), newLit(2)), newLit(4))) - --> col 2: $MODULE.eval - col 3: proc (PPlusExpr): int - col 7: "" + ```nim + method eval(e: PExpr): int = quit "to override!" + method eval(e: PLiteral): int = e.x + method eval(e: PPlusExpr): int = eval(e.a) + eval(e.b) + echo eval(newPlus(newPlus(newLit(1), newLit(2)), newLit(4))) + --> col 2: $MODULE.eval + col 3: proc (PPlusExpr): int + col 7: "" + ``` skParam ------- -| **Third column**: module + [n scope nesting] + param name. +| **Third column**: module + \[n scope nesting] + param name. | **Fourth column**: the type of the parameter. | **Docstring**: always the empty string. -.. code-block:: nim - proc reader(filename = "tests.nim") = - let text = readFile(filename) - --> col 2: $MODULE.reader.filename - col 3: string - col 7: "" + ```nim + proc reader(filename = "tests.nim") = + let text = readFile(filename) + --> col 2: $MODULE.reader.filename + col 3: string + col 7: "" + ``` skProc @@ -419,34 +435,36 @@ While at the language level a proc is differentiated from others by the parameters and return value, the signature of the proc returned by idetools returns also the pragmas for the proc. -| **Third column**: module + [n scope nesting] + proc name. +| **Third column**: module + \[n scope nesting] + proc name. | **Fourth column**: signature of the proc including return type. | **Docstring**: docstring if available. -.. code-block:: nim - open(filename, fmWrite) - --> col 2: system.Open - col 3: proc (var File, string, FileMode, int): bool - col 7: - "Opens a file named `filename` with given `mode`. + ```nim + open(filename, fmWrite) + --> col 2: system.Open + col 3: proc (var File, string, FileMode, int): bool + col 7: + "Opens a file named `filename` with given `mode`. - Default mode is readonly. Returns true iff the file could be opened. - This throws no exception if the file could not be opened." + Default mode is readonly. Returns true iff the file could be opened. + This throws no exception if the file could not be opened." + ``` skResult -------- -| **Third column**: module + [n scope nesting] + result. +| **Third column**: module + \[n scope nesting] + result. | **Fourth column**: the type of the result. | **Docstring**: always the empty string. -.. code-block:: nim - proc getRandomValue() : int = - return 4 - --> col 2: $MODULE.getRandomValue.result - col 3: int - col 7: "" + ```nim + proc getRandomValue() : int = + return 4 + --> col 2: $MODULE.getRandomValue.result + col 3: int + col 7: "" + ``` skTemplate @@ -457,11 +475,11 @@ defined, since at that point in the file the parser hasn't processed the full line yet. The signature will be returned complete in posterior instances of the template. -| **Third column**: module + [n scope nesting] + template name. +| **Third column**: module + \[n scope nesting] + template name. | **Fourth column**: signature of the template including return type. | **Docstring**: docstring if available. -.. code-block:: nim + `````nim let text = "some text" letters = toSeq(runes(text)) @@ -472,52 +490,56 @@ posterior instances of the template. Example: - .. code-block:: nim + ```nim let numeric = @[1, 2, 3, 4, 5, 6, 7, 8, 9] odd_numbers = toSeq(filter(numeric) do (x: int) -> bool: if x mod 2 == 1: result = true) assert odd_numbers == @[1, 3, 5, 7, 9]" + ``` + ````` skType ------ -| **Third column**: module + [n scope nesting] + type name. +| **Third column**: module + \[n scope nesting] + type name. | **Fourth column**: the type. | **Docstring**: always the empty string. -.. code-block:: nim - proc writeTempFile() = - var output: File - --> col 2: system.File - col 3: File - col 7: "" + ```nim + proc writeTempFile() = + var output: File + --> col 2: system.File + col 3: File + col 7: "" + ``` skVar ----- -| **Third column**: module + [n scope nesting] + var name. +| **Third column**: module + \[n scope nesting] + var name. | **Fourth column**: the type of the var. | **Docstring**: always the empty string. -.. code-block:: nim - proc writeTempFile() = - var output: File - output.open("/tmp/somefile", fmWrite) - output.write("test") - --> col 2: $MODULE.writeTempFile.output - col 3: File - col 7: "" + ```nim + proc writeTempFile() = + var output: File + output.open("/tmp/somefile", fmWrite) + output.write("test") + --> col 2: $MODULE.writeTempFile.output + col 3: File + col 7: "" + ``` Test suite ========== To verify that idetools is working properly there are files in the -``tests/caas/`` directory which provide unit testing. If you find +`tests/caas/` directory which provide unit testing. If you find odd idetools behaviour and are able to reproduce it, you are welcome to report it as a bug and add a test to the suite to avoid future regressions. @@ -528,55 +550,59 @@ Running the test suite At the moment idetools support is still in development so the test suite is not integrated with the main test suite and you have to -run it manually. First you have to compile the tester:: +run it manually. First you have to compile the tester: - $ cd my/nim/checkout/tests - $ nim c testament/caasdriver.nim + ```cmd + $ cd my/nim/checkout/tests + $ nim c testament/caasdriver.nim + ``` -Running the ``caasdriver`` without parameters will attempt to process +Running the `caasdriver` without parameters will attempt to process all the test cases in all three operation modes. If a test succeeds nothing will be printed and the process will exit with zero. If any test fails, the specific line of the test preceding the failure and the failure itself will be dumped to stdout, along with a final indicator of the success state and operation mode. You can pass the -parameter ``verbose`` to force all output even on successful tests. +parameter `verbose` to force all output even on successful tests. -The normal operation mode is called ``ProcRun`` and it involves +The normal operation mode is called `ProcRun` and it involves starting a process for each command or query, similar to running -manually the Nim compiler from the commandline. The ``CaasRun`` -mode starts a server process to answer all queries. The ``SymbolProcRun`` +manually the Nim compiler from the commandline. The `CaasRun` +mode starts a server process to answer all queries. The `SymbolProcRun` mode is used by compiler developers. This means that running all -tests involves processing all ``*.txt`` files three times, which +tests involves processing all `*.txt` files three times, which can be quite time consuming. If you don't want to run all the test case files you can pass any -substring as a parameter to ``caasdriver``. Only files matching the +substring as a parameter to `caasdriver`. Only files matching the passed substring will be run. The filtering doesn't use any globbing metacharacters, it's a plain match. For example, to run only -``*-compile*.txt`` tests in verbose mode:: +`*-compile*.txt` tests in verbose mode: - ./caasdriver verbose -compile + ```cmd + ./caasdriver verbose -compile + ``` Test case file format --------------------- -All the ``tests/caas/*.txt`` files encode a session with the compiler: +All the `tests/caas/*.txt` files encode a session with the compiler: * The first line indicates the main project file. -* Lines starting with ``>`` indicate a command to be sent to the +* Lines starting with `>` indicate a command to be sent to the compiler and the lines following a command include checks for - expected or forbidden output (``!`` for forbidden). + expected or forbidden output (`!` for forbidden). -* If a line starts with ``#`` it will be ignored completely, so you +* If a line starts with `#` it will be ignored completely, so you can use that for comments. -* Since some cases are specific to either ``ProcRun`` or ``CaasRun`` +* Since some cases are specific to either `ProcRun` or `CaasRun` modes, you can prefix a line with the mode and the line will be processed only in that mode. -* The rest of the line is treated as a `regular expression <re.html>`_, +* The rest of the line is treated as a [regular expression](re.html), so be careful escaping metacharacters like parenthesis. Before the line is processed as a regular expression, some basic diff --git a/doc/intern.md b/doc/intern.md new file mode 100644 index 000000000..6b16bc71f --- /dev/null +++ b/doc/intern.md @@ -0,0 +1,679 @@ +========================================= + Internals of the Nim Compiler +========================================= + + +:Author: Andreas Rumpf +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +> "Abstraction is layering ignorance on top of reality." -- Richard Gabriel + + +Directory structure +=================== + +The Nim project's directory structure is: + +============ =================================================== +Path Purpose +============ =================================================== +`bin` generated binary files +`build` generated C code for the installation +`compiler` the Nim compiler itself; note that this + code has been translated from a bootstrapping + version written in Pascal, so the code is **not** + a poster child of good Nim code +`config` configuration files for Nim +`dist` additional packages for the distribution +`doc` the documentation; it is a bunch of + reStructuredText files +`lib` the Nim library +============ =================================================== + + +Bootstrapping the compiler +========================== + +**Note**: Add ``.`` to your PATH so that `koch`:cmd: can be used without the ``./``. + +Compiling the compiler is a simple matter of running: + + ```cmd + nim c koch.nim + koch boot -d:release + ``` + +For a debug version use: + + ```cmd + nim c koch.nim + koch boot + ``` + + +And for a debug version compatible with GDB: + + ```cmd + nim c koch.nim + koch boot --debuginfo --linedir:on + ``` + +The `koch`:cmd: program is Nim's maintenance script. It is a replacement for +make and shell scripting with the advantage that it is much more portable. +More information about its options can be found in the [koch](koch.html) +documentation. + + +Reproducible builds +------------------- + +Set the compilation timestamp with the `SOURCE_DATE_EPOCH` environment variable. + + ```cmd + export SOURCE_DATE_EPOCH=$(git log -n 1 --format=%at) + koch boot # or `./build_all.sh` + ``` + + +Debugging the compiler +====================== + + +Bisecting for regressions +------------------------- + +There are often times when there is a bug that is caused by a regression in the +compiler or stdlib. Bisecting the Nim repo commits is a useful tool to identify +what commit introduced the regression. + +Even if it's not known whether a bug is caused by a regression, bisection can reduce +debugging time by ruling it out. If the bug is found to be a regression, then you +focus on the changes introduced by that one specific commit. + +`koch temp`:cmd: returns 125 as the exit code in case the compiler +compilation fails. This exit code tells `git bisect`:cmd: to skip the +current commit: + + ```cmd + git bisect start bad-commit good-commit + git bisect run ./koch temp -r c test-source.nim + ``` + +You can also bisect using custom options to build the compiler, for example if +you don't need a debug version of the compiler (which runs slower), you can replace +`./koch temp`:cmd: by explicit compilation command, see [Bootstrapping the compiler]. + +See also: + +- Crossplatform C/Cpp/Valgrind/JS Bisect in GitHub: https://github.com/juancarlospaco/nimrun-action#examples + + +Building an instrumented compiler +--------------------------------- + +Considering that a useful method of debugging the compiler is inserting debug +logging, or changing code and then observing the outcome of a testcase, it is +fastest to build a compiler that is instrumented for debugging from an +existing release build. `koch temp`:cmd: provides a convenient method of doing +just that. + +By default, running `koch temp`:cmd: will build a lean version of the compiler +with `-d:debug`:option: enabled. The compiler is written to `bin/nim_temp` by +default. A lean version of the compiler lacks JS and documentation generation. + +`bin/nim_temp` can be directly used to run testcases, or used with testament +with `testament --nim:bin/nim_temp r tests/category/tsometest`:cmd:. + +`koch temp`:cmd: will build the temporary compiler with the `-d:debug`:option: +enabled. Here are compiler options that are of interest when debugging: + +* `-d:debug`:option:\: enables `assert` statements and stacktraces and all + runtime checks +* `--opt:speed`:option:\: build with optimizations enabled +* `--debugger:native`:option:\: enables `--debuginfo --lineDir:on`:option: for using + a native debugger like GDB, LLDB or CDB +* `-d:nimDebug`:option: cause calls to `quit` to raise an assertion exception +* `-d:nimDebugUtils`:option:\: enables various debugging utilities; + see `compiler/debugutils` +* `-d:stacktraceMsgs -d:nimCompilerStacktraceHints`:option:\: adds some additional + stacktrace hints; see https://github.com/nim-lang/Nim/pull/13351 +* `-u:leanCompiler`:option:\: enable JS and doc generation + +Another method to build and run the compiler is directly through `koch`:cmd:\: + + ```cmd + koch temp [options] c test.nim + + # (will build with js support) + koch temp [options] js test.nim + + # (will build with doc support) + koch temp [options] doc test.nim + ``` + +Debug logging +------------- + +"Printf debugging" is still the most appropriate way to debug many problems +arising in compiler development. The typical usage of breakpoints to debug +the code is often less practical, because almost all code paths in the +compiler will be executed hundreds of times before a particular section of the +tested program is reached where the newly developed code must be activated. + +To work around this problem, you'll typically introduce an if statement in the +compiler code detecting more precisely the conditions where the tested feature +is being used. One very common way to achieve this is to use the `mdbg` condition, +which will be true only in contexts, processing expressions and statements from +the currently compiled main module: + + ```nim + # inside some compiler module + if mdbg: + debug someAstNode + ``` + +Using the `isCompilerDebug`:nim: condition along with inserting some statements +into the testcase provides more granular logging: + + ```nim + # compilermodule.nim + if isCompilerDebug(): + debug someAstNode + + # testcase.nim + proc main = + {.define(nimCompilerDebug).} + let a = 2.5 * 3 + {.undef(nimCompilerDebug).} + ``` + +Logging can also be scoped to a specific filename as well. This will of course +match against every module with that name. + + ```nim + if `??`(conf, n.info, "module.nim"): + debug(n) + ``` + +The above examples also makes use of the `debug`:nim: proc, which is able to +print a human-readable form of an arbitrary AST tree. Other common ways to print +information about the internal compiler types include: + + ```nim + # pretty print PNode + + # pretty prints the Nim ast + echo renderTree(someNode) + + # pretty prints the Nim ast, but annotates symbol IDs + echo renderTree(someNode, {renderIds}) + + # pretty print ast as JSON + debug(someNode) + + # print as YAML + echo treeToYaml(config, someNode) + + + # pretty print PType + + # print type name + echo typeToString(someType) + + # pretty print as JSON + debug(someType) + + # print as YAML + echo typeToYaml(config, someType) + + + # pretty print PSym + + # print the symbol's name + echo symbol.name.s + + # pretty print as JSON + debug(symbol) + + # print as YAML + echo symToYaml(config, symbol) + + + # pretty print TLineInfo + lineInfoToStr(lineInfo) + + + # print the structure of any type + repr(someVar) + ``` + +Here are some other helpful utilities: + + ```nim + # how did execution reach this location? + writeStackTrace() + ``` + +These procs may not already be imported by the module you're editing. +You can import them directly for debugging: + + ```nim + from astalgo import debug + from types import typeToString + from renderer import renderTree + from msgs import `??` + ``` + +Native debugging +---------------- + +Stepping through the compiler with a native debugger is a very powerful tool to +both learn and debug it. However, there is still the need to constrain when +breakpoints are triggered. The same methods as in [Debug logging] can be applied +here when combined with calls to the debug helpers `enteringDebugSection()`:nim: +and `exitingDebugSection()`:nim:. + +#. Compile the temp compiler with `--debugger:native -d:nimDebugUtils`:option: +#. Set your desired breakpoints or watchpoints. +#. Configure your debugger: + * GDB: execute `source tools/compiler.gdb` at startup + * LLDB execute `command source tools/compiler.lldb` at startup +#. Use one of the scoping helpers like so: + + ```nim + if isCompilerDebug(): + enteringDebugSection() + else: + exitingDebugSection() + ``` + +A caveat of this method is that all breakpoints and watchpoints are enabled or +disabled. Also, due to a bug, only breakpoints can be constrained for LLDB. + +The compiler's architecture +=========================== + +Nim uses the classic compiler architecture: A lexer/scanner feeds tokens to a +parser. The parser builds a syntax tree that is used by the code generators. +This syntax tree is the interface between the parser and the code generator. +It is essential to understand most of the compiler's code. + +Semantic analysis is separated from parsing. + +.. include:: filelist.txt + + +The syntax tree +--------------- +The syntax tree consists of nodes which may have an arbitrary number of +children. Types and symbols are represented by other nodes, because they +may contain cycles. The AST changes its shape after semantic checking. This +is needed to make life easier for the code generators. See the "ast" module +for the type definitions. The [macros](macros.html) module contains many +examples how the AST represents each syntactic structure. + + +Runtimes +======== + +Nim has two different runtimes, the "old runtime" and the "new runtime". The old +runtime supports the old GCs (markAndSweep, refc, Boehm), the new runtime supports +ARC/ORC. The new runtime is active `when defined(nimV2)`. + + +Coding Guidelines +================= + +* We follow Nim's official style guide, see [NEP1](nep1.html). +* Max line length is 100 characters. +* Provide spaces around binary operators if that enhances readability. +* Use a space after a colon, but not before it. +* (deprecated) Start types with a capital `T`, unless they are + pointers/references which start with `P`. +* Prefer `import package`:nim: over `from package import symbol`:nim:. + +See also the [API naming design](apis.html) document. + + +Porting to new platforms +======================== + +Porting Nim to a new architecture is pretty easy, since C is the most +portable programming language (within certain limits) and Nim generates +C code, porting the code generator is not necessary. + +POSIX-compliant systems on conventional hardware are usually pretty easy to +port: Add the platform to `platform` (if it is not already listed there), +check that the OS, System modules work and recompile Nim. + +The only case where things aren't as easy is when old runtime's garbage +collectors need some assembler tweaking to work. The default +implementation uses C's `setjmp`:c: function to store all registers +on the hardware stack. It may be necessary that the new platform needs to +replace this generic code by some assembler code. + +Files that may need changed for your platform include: + +* `compiler/platform.nim` + Add os/cpu properties. +* `lib/system.nim` + Add os/cpu to the documentation for `system.hostOS` and `system.hostCPU`. +* `compiler/options.nim` + Add special os/cpu property checks in `isDefined`. +* `compiler/installer.ini` + Add os/cpu to `Project.Platforms` field. +* `lib/system/platforms.nim` + Add os/cpu. +* `std/private/osseps.nim` + Add os specializations. +* `lib/pure/distros.nim` + Add os, package handler. +* `tools/niminst/makefile.nimf` + Add os/cpu compiler/linker flags. +* `tools/niminst/buildsh.nimf` + Add os/cpu compiler/linker flags. + +If the `--os` or `--cpu` options aren't passed to the compiler, then Nim will +determine the current host os, cpu and endianness from `system.cpuEndian`, +`system.hostOS` and `system.hostCPU`. Those values are derived from +`compiler/platform.nim`. + +In order for the new platform to be bootstrapped from the `csources`, it must: + +* have `compiler/platform.nim` updated +* have `compiler/installer.ini` updated +* have `tools/niminst/buildsh.nimf` updated +* have `tools/niminst/makefile.nimf` updated +* be backported to the Nim version used by the `csources` +* the new `csources` must be pushed +* the new `csources` revision must be updated in `config/build_config.txt` + + +Runtime type information +======================== + +**Note**: This section describes the "old runtime". + +*Runtime type information* (RTTI) is needed for several aspects of the Nim +programming language: + +Garbage collection +: The old GCs use the RTTI for traversing arbitrary Nim types, but usually + only the `marker` field which contains a proc that does the traversal. + +Complex assignments +: Sequences and strings are implemented as + pointers to resizable buffers, but Nim requires copying for + assignments. Apart from RTTI the compiler also generates copy procedures + as a specialization. + +We already know the type information as a graph in the compiler. +Thus, we need to serialize this graph as RTTI for C code generation. +Look at the file ``lib/system/hti.nim`` for more information. + + +Magics and compilerProcs +======================== + +The `system` module contains the part of the RTL which needs support by +compiler magic. The C code generator generates the C code for it, just like any other +module. However, calls to some procedures like `addInt` are inserted by +the generator. Therefore, there is a table (`compilerprocs`) +with all symbols that are marked as `compilerproc`. `compilerprocs` are +needed by the code generator. A `magic` proc is not the same as a +`compilerproc`: A `magic` is a proc that needs compiler magic for its +semantic checking, a `compilerproc` is a proc that is used by the code +generator. + + +Code generation for closures +============================ + +Code generation for closures is implemented by `lambda lifting`:idx:. + + +Design +------ + +A `closure` proc var can call ordinary procs of the default Nim calling +convention. But not the other way round! A closure is implemented as a +`tuple[prc, env]`. `env` can be nil implying a call without a closure. +This means that a call through a closure generates an `if` but the +interoperability is worth the cost of the `if`. Thunk generation would be +possible too, but it's slightly more effort to implement. + +Tests with GCC on Amd64 showed that it's really beneficial if the +'environment' pointer is passed as the last argument, not as the first argument. + +Proper thunk generation is harder because the proc that is to wrap +could stem from a complex expression: + + ```nim + receivesClosure(returnsDefaultCC[i]) + ``` + +A thunk would need to call `returnsDefaultCC[i]` somehow and that would require +an *additional* closure generation... Ok, not really, but it requires to pass +the function to call. So we'd end up with 2 indirect calls instead of one. +Another much more severe problem with this solution is that it's not GC-safe +to pass a proc pointer around via a generic `ref` type. + + +Example code: + + ```nim + proc add(x: int): proc (y: int): int {.closure.} = + return proc (y: int): int = + return x + y + + var add2 = add(2) + echo add2(5) #OUT 7 + ``` + +This should produce roughly this code: + + ```nim + type + Env = ref object + x: int # data + + proc anon(y: int, c: Env): int = + return y + c.x + + proc add(x: int): tuple[prc, data] = + var env: Env + new env + env.x = x + result = (anon, env) + + var add2 = add(2) + let tmp = if add2.data == nil: add2.prc(5) else: add2.prc(5, add2.data) + echo tmp + ``` + + +Beware of nesting: + + ```nim + proc add(x: int): proc (y: int): proc (z: int): int {.closure.} {.closure.} = + return lambda (y: int): proc (z: int): int {.closure.} = + return lambda (z: int): int = + return x + y + z + + var add24 = add(2)(4) + echo add24(5) #OUT 11 + ``` + +This should produce roughly this code: + + ```nim + type + EnvX = ref object + x: int # data + + EnvY = ref object + y: int + ex: EnvX + + proc lambdaZ(z: int, ey: EnvY): int = + return ey.ex.x + ey.y + z + + proc lambdaY(y: int, ex: EnvX): tuple[prc, data: EnvY] = + var ey: EnvY + new ey + ey.y = y + ey.ex = ex + result = (lambdaZ, ey) + + proc add(x: int): tuple[prc, data: EnvX] = + var ex: EnvX + ex.x = x + result = (lambdaY, ex) + + var tmp = add(2) + var tmp2 = tmp.fn(4, tmp.data) + var add24 = tmp2.fn(4, tmp2.data) + echo add24(5) + ``` + + +We could get rid of nesting environments by always inlining inner anon procs. +More useful is escape analysis and stack allocation of the environment, +however. + + +Accumulator +----------- + + ```nim + proc getAccumulator(start: int): proc (): int {.closure} = + var i = start + return lambda: int = + inc i + return i + + proc p = + var delta = 7 + proc accumulator(start: int): proc(): int = + var x = start-1 + result = proc (): int = + x = x + delta + inc delta + return x + + var a = accumulator(3) + var b = accumulator(4) + echo a() + b() + ``` + + +Internals +--------- + +Lambda lifting is implemented as part of the `transf` pass. The `transf` +pass generates code to set up the environment and to pass it around. However, +this pass does not change the types! So we have some kind of mismatch here; on +the one hand the proc expression becomes an explicit tuple, on the other hand +the tyProc(ccClosure) type is not changed. For C code generation it's also +important the hidden formal param is `void*`:c: and not something more +specialized. However, the more specialized env type needs to passed to the +backend somehow. We deal with this by modifying `s.ast[paramPos]` to contain +the formal hidden parameter, but not `s.typ`! + + +Notes on type and AST representation +==================================== + +To be expanded. + + +Integer literals +---------------- + +In Nim, there is a redundant way to specify the type of an +integer literal. First, it should be unsurprising that every +node has a node kind. The node of an integer literal can be any of the +following values: + + nkIntLit, nkInt8Lit, nkInt16Lit, nkInt32Lit, nkInt64Lit, + nkUIntLit, nkUInt8Lit, nkUInt16Lit, nkUInt32Lit, nkUInt64Lit + +On top of that, there is also the `typ` field for the type. The +kind of the `typ` field can be one of the following ones, and it +should be matching the literal kind: + + tyInt, tyInt8, tyInt16, tyInt32, tyInt64, tyUInt, tyUInt8, + tyUInt16, tyUInt32, tyUInt64 + +Then there is also the integer literal type. This is a specific type +that is implicitly convertible into the requested type if the +requested type can hold the value. For this to work, the type needs to +know the concrete value of the literal. For example an expression +`321` will be of type `int literal(321)`. This type is implicitly +convertible to all integer types and ranges that contain the value +`321`. That would be all builtin integer types except `uint8` and +`int8` where `321` would be out of range. When this literal type is +assigned to a new `var` or `let` variable, it's type will be resolved +to just `int`, not `int literal(321)` unlike constants. A constant +keeps the full `int literal(321)` type. Here is an example where that +difference matters. + + + ```nim + proc foo(arg: int8) = + echo "def" + + const tmp1 = 123 + foo(tmp1) # OK + + let tmp2 = 123 + foo(tmp2) # Error + ``` + +In a context with multiple overloads, the integer literal kind will +always prefer the `int` type over all other types. If none of the +overloads is of type `int`, then there will be an error because of +ambiguity. + + ```nim + proc foo(arg: int) = + echo "abc" + proc foo(arg: int8) = + echo "def" + foo(123) # output: abc + + proc bar(arg: int16) = + echo "abc" + proc bar(arg: int8) = + echo "def" + + bar(123) # Error ambiguous call + ``` + +In the compiler these integer literal types are represented with the +node kind `nkIntLit`, type kind `tyInt` and the member `n` of the type +pointing back to the integer literal node in the ast containing the +integer value. These are the properties that hold true for integer +literal types. + + n.kind == nkIntLit + n.typ.kind == tyInt + n.typ.n == n + +Other literal types, such as `uint literal(123)` that would +automatically convert to other integer types, but prefers to +become a `uint` are not part of the Nim language. + +In an unchecked AST, the `typ` field is nil. The type checker will set +the `typ` field accordingly to the node kind. Nodes of kind `nkIntLit` +will get the integer literal type (e.g. `int literal(123)`). Nodes of +kind `nkUIntLit` will get type `uint` (kind `tyUint`), etc. + +This also means that it is not possible to write a literal in an +unchecked AST that will after sem checking just be of type `int` and +not implicitly convertible to other integer types. This only works for +all integer types that are not `int`. diff --git a/doc/intern.rst b/doc/intern.rst deleted file mode 100644 index be183fd8e..000000000 --- a/doc/intern.rst +++ /dev/null @@ -1,786 +0,0 @@ -========================================= - Internals of the Nim Compiler -========================================= - - -:Author: Andreas Rumpf -:Version: |nimversion| - -.. contents:: - - "Abstraction is layering ignorance on top of reality." -- Richard Gabriel - - -Directory structure -=================== - -The Nim project's directory structure is: - -============ =================================================== -Path Purpose -============ =================================================== -``bin`` generated binary files -``build`` generated C code for the installation -``compiler`` the Nim compiler itself; note that this - code has been translated from a bootstrapping - version written in Pascal, so the code is **not** - a poster child of good Nim code -``config`` configuration files for Nim -``dist`` additional packages for the distribution -``doc`` the documentation; it is a bunch of - reStructuredText files -``lib`` the Nim library -``web`` website of Nim; generated by ``nimweb`` - from the ``*.txt`` and ``*.nimf`` files -============ =================================================== - - -Bootstrapping the compiler -========================== - -Compiling the compiler is a simple matter of running:: - - nim c koch.nim - ./koch boot - -For a release version use:: - - nim c koch.nim - ./koch boot -d:release - -And for a debug version compatible with GDB:: - - nim c koch.nim - ./koch boot --debuginfo --linedir:on - -The ``koch`` program is Nim's maintenance script. It is a replacement for -make and shell scripting with the advantage that it is much more portable. -More information about its options can be found in the `koch <koch.html>`_ -documentation. - - -Coding Guidelines -================= - -* Use CamelCase, not underscored_identifiers. -* Indent with two spaces. -* Max line length is 80 characters. -* Provide spaces around binary operators if that enhances readability. -* Use a space after a colon, but not before it. -* [deprecated] Start types with a capital ``T``, unless they are - pointers/references which start with ``P``. - -See also the `API naming design <apis.html>`_ document. - - -Porting to new platforms -======================== - -Porting Nim to a new architecture is pretty easy, since C is the most -portable programming language (within certain limits) and Nim generates -C code, porting the code generator is not necessary. - -POSIX-compliant systems on conventional hardware are usually pretty easy to -port: Add the platform to ``platform`` (if it is not already listed there), -check that the OS, System modules work and recompile Nim. - -The only case where things aren't as easy is when the garbage -collector needs some assembler tweaking to work. The standard -version of the GC uses C's ``setjmp`` function to store all registers -on the hardware stack. It may be necessary that the new platform needs to -replace this generic code by some assembler code. - - -Runtime type information -======================== - -*Runtime type information* (RTTI) is needed for several aspects of the Nim -programming language: - -Garbage collection - The most important reason for RTTI. Generating - traversal procedures produces bigger code and is likely to be slower on - modern hardware as dynamic procedure binding is hard to predict. - -Complex assignments - Sequences and strings are implemented as - pointers to resizeable buffers, but Nim requires copying for - assignments. Apart from RTTI the compiler could generate copy procedures - for any type that needs one. However, this would make the code bigger and - the RTTI is likely already there for the GC. - -We already know the type information as a graph in the compiler. -Thus we need to serialize this graph as RTTI for C code generation. -Look at the file ``lib/system/hti.nim`` for more information. - -Rebuilding the compiler -======================== - -After an initial build via `sh build_all.sh` on posix or `build_all.bat` on windows, -you can rebuild the compiler as follows: -* `nim c koch` if you need to rebuild koch -* `./koch boot -d:release` this ensures the compiler can rebuild itself - (use `koch` instead of `./koch` on windows), which builds the compiler 3 times. - -A faster approach if you don't need to run the full bootstrapping implied by `koch boot`, -is the following: -* `pathto/nim c --lib:lib -d:release -o:bin/nim_temp compiler/nim.nim` -Where `pathto/nim` is any nim binary sufficiently recent (eg `bin/nim_cources` -built during bootstrap or `$HOME/.nimble/bin/nim` installed by `choosenim 1.2.0`) - -You can pass any additional options such as `-d:leanCompiler` if you don't need -certain features or `-d:debug --stacktrace:on --excessiveStackTrace --stackTraceMsgs` -for debugging the compiler. See also -[Debugging the compiler](intern.html#debugging-the-compiler). - -Debugging the compiler -====================== - -You can of course use GDB or Visual Studio to debug the -compiler (via ``--debuginfo --lineDir:on``). However, there -are also lots of procs that aid in debugging: - - -.. code-block:: nim - # pretty prints the Nim AST - echo renderTree(someNode) - # outputs some JSON representation - debug(someNode) - # pretty prints some type - echo typeToString(someType) - debug(someType) - echo symbol.name.s - debug(symbol) - # pretty prints the Nim ast, but annotates symbol IDs: - echo renderTree(someNode, {renderIds}) - if n.info ?? "temp.nim": - # only output when it comes from "temp.nim" - echo renderTree(n) - if n.info ?? "temp.nim": - # why does it process temp.nim here? - writeStackTrace() - -To create a new compiler for each run, use ``koch temp``:: - - ./koch temp c /tmp/test.nim - -``koch temp`` creates a debug build of the compiler, which is useful -to create stacktraces for compiler debugging. See also -[Rebuilding the compiler](intern.html#rebuilding-the-compiler) if you need -more control. - -Bisecting for regressions -========================= - -``koch temp`` returns 125 as the exit code in case the compiler -compilation fails. This exit code tells ``git bisect`` to skip the -current commit.:: - - git bisect start bad-commit good-commit - git bisect run ./koch temp -r c test-source.nim - -You can also bisect using custom options to build the compiler, for example if -you don't need a debug version of the compiler (which runs slower), you can replace -`./koch temp` by explicit compilation command, see -[Rebuilding the compiler](intern.html#rebuilding-the-compiler). - -The compiler's architecture -=========================== - -Nim uses the classic compiler architecture: A lexer/scanner feds tokens to a -parser. The parser builds a syntax tree that is used by the code generator. -This syntax tree is the interface between the parser and the code generator. -It is essential to understand most of the compiler's code. - -In order to compile Nim correctly, type-checking has to be separated from -parsing. Otherwise generics cannot work. - -.. include:: filelist.txt - - -The syntax tree ---------------- -The syntax tree consists of nodes which may have an arbitrary number of -children. Types and symbols are represented by other nodes, because they -may contain cycles. The AST changes its shape after semantic checking. This -is needed to make life easier for the code generators. See the "ast" module -for the type definitions. The `macros <macros.html>`_ module contains many -examples how the AST represents each syntactic structure. - - -How the RTL is compiled -======================= - -The ``system`` module contains the part of the RTL which needs support by -compiler magic (and the stuff that needs to be in it because the spec -says so). The C code generator generates the C code for it, just like any other -module. However, calls to some procedures like ``addInt`` are inserted by -the CCG. Therefore the module ``magicsys`` contains a table (``compilerprocs``) -with all symbols that are marked as ``compilerproc``. ``compilerprocs`` are -needed by the code generator. A ``magic`` proc is not the same as a -``compilerproc``: A ``magic`` is a proc that needs compiler magic for its -semantic checking, a ``compilerproc`` is a proc that is used by the code -generator. - - -Compilation cache -================= - -The implementation of the compilation cache is tricky: There are lots -of issues to be solved for the front- and backend. - - -General approach: AST replay ----------------------------- - -We store a module's AST of a successful semantic check in a SQLite -database. There are plenty of features that require a sub sequence -to be re-applied, for example: - -.. code-block:: nim - {.compile: "foo.c".} # even if the module is loaded from the DB, - # "foo.c" needs to be compiled/linked. - -The solution is to **re-play** the module's top level statements. -This solves the problem without having to special case the logic -that fills the internal seqs which are affected by the pragmas. - -In fact, this describes how the AST should be stored in the database, -as a "shallow" tree. Let's assume we compile module ``m`` with the -following contents: - -.. code-block:: nim - import strutils - - var x*: int = 90 - {.compile: "foo.c".} - proc p = echo "p" - proc q = echo "q" - static: - echo "static" - -Conceptually this is the AST we store for the module: - -.. code-block:: nim - import strutils - - var x* - {.compile: "foo.c".} - proc p - proc q - static: - echo "static" - -The symbol's ``ast`` field is loaded lazily, on demand. This is where most -savings come from, only the shallow outer AST is reconstructed immediately. - -It is also important that the replay involves the ``import`` statement so -that dependencies are resolved properly. - - -Shared global compiletime state -------------------------------- - -Nim allows ``.global, compiletime`` variables that can be filled by macro -invocations across different modules. This feature breaks modularity in a -severe way. Plenty of different solutions have been proposed: - -- Restrict the types of global compiletime variables to ``Set[T]`` or - similar unordered, only-growable collections so that we can track - the module's write effects to these variables and reapply the changes - in a different order. -- In every module compilation, reset the variable to its default value. -- Provide a restrictive API that can load/save the compiletime state to - a file. - -(These solutions are not mutually exclusive.) - -Since we adopt the "replay the top level statements" idea, the natural -solution to this problem is to emit pseudo top level statements that -reflect the mutations done to the global variable. However, this is -MUCH harder than it sounds, for example ``squeaknim`` uses this -snippet: - -.. code-block:: nim - apicall.add(") module: '" & dllName & "'>\C" & - "\t^self externalCallFailed\C!\C\C") - stCode.add(st & "\C\t\"Generated by NimSqueak\"\C\t" & apicall) - -We can "replay" ``stCode.add`` only if the values of ``st`` -and ``apicall`` are known. And even then a hash table's ``add`` with its -hashing mechanism is too hard to replay. - -In practice, things are worse still, consider ``someGlobal[i][j].add arg``. -We only know the root is ``someGlobal`` but the concrete path to the data -is unknown as is the value that is added. We could compute a "diff" between -the global states and use that to compute a symbol patchset, but this is -quite some work, expensive to do at runtime (it would need to run after -every module has been compiled) and would also break for hash tables. - -We need an API that hides the complex aliasing problems by not relying -on Nim's global variables. The obvious solution is to use string keys -instead of global variables: - -.. code-block:: nim - - proc cachePut*(key: string; value: string) - proc cacheGet*(key: string): string - -However, the values being strings/json is quite problematic: Many -lookup tables that are built at compiletime embed *proc vars* and -types which have no obvious string representation... Seems like -AST diffing is still the best idea as it will not require to use -an alien API and works with some existing Nimble packages, at least. - -On the other hand, in Nim's future I would like to replace the VM -by native code. A diff algorithm wouldn't work for that. -Instead the native code would work with an API like ``put``, ``get``: - -.. code-block:: nim - - proc cachePut*(key: string; value: NimNode) - proc cacheGet*(key: string): NimNode - -The API should embrace the AST diffing notion: See the -module ``macrocache`` for the final details. - - - -Methods and type converters ---------------------------- - -In the following -sections *global* means *shared between modules* or *property of the whole -program*. - -Nim contains language features that are *global*. The best example for that -are multi methods: Introducing a new method with the same name and some -compatible object parameter means that the method's dispatcher needs to take -the new method into account. So the dispatching logic is only completely known -after the whole program has been translated! - -Other features that are *implicitly* triggered cause problems for modularity -too. Type converters fall into this category: - -.. code-block:: nim - # module A - converter toBool(x: int): bool = - result = x != 0 - -.. code-block:: nim - # module B - import A - - if 1: - echo "ugly, but should work" - -If in the above example module ``B`` is re-compiled, but ``A`` is not then -``B`` needs to be aware of ``toBool`` even though ``toBool`` is not referenced -in ``B`` *explicitly*. - -Both the multi method and the type converter problems are solved by the -AST replay implementation. - - -Generics -~~~~~~~~ - -We cache generic instantiations and need to ensure this caching works -well with the incremental compilation feature. Since the cache is -attached to the ``PSym`` datastructure, it should work without any -special logic. - - -Backend issues --------------- - -- Init procs must not be "forgotten" to be called. -- Files must not be "forgotten" to be linked. -- Method dispatchers are global. -- DLL loading via ``dlsym`` is global. -- Emulated thread vars are global. - -However the biggest problem is that dead code elimination breaks modularity! -To see why, consider this scenario: The module ``G`` (for example the huge -Gtk2 module...) is compiled with dead code elimination turned on. So none -of ``G``'s procs is generated at all. - -Then module ``B`` is compiled that requires ``G.P1``. Ok, no problem, -``G.P1`` is loaded from the symbol file and ``G.c`` now contains ``G.P1``. - -Then module ``A`` (that depends on ``B`` and ``G``) is compiled and ``B`` -and ``G`` are left unchanged. ``A`` requires ``G.P2``. - -So now ``G.c`` MUST contain both ``P1`` and ``P2``, but we haven't even -loaded ``P1`` from the symbol file, nor do we want to because we then quickly -would restore large parts of the whole program. - - -Solution -~~~~~~~~ - -The backend must have some logic so that if the currently processed module -is from the compilation cache, the ``ast`` field is not accessed. Instead -the generated C(++) for the symbol's body needs to be cached too and -inserted back into the produced C file. This approach seems to deal with -all the outlined problems above. - - -Debugging Nim's memory management -================================= - -The following paragraphs are mostly a reminder for myself. Things to keep -in mind: - -* If an assertion in Nim's memory manager or GC fails, the stack trace - keeps allocating memory! Thus a stack overflow may happen, hiding the - real issue. -* What seem to be C code generation problems is often a bug resulting from - not producing prototypes, so that some types default to ``cint``. Testing - without the ``-w`` option helps! - - -The Garbage Collector -===================== - -Introduction ------------- - -I use the term *cell* here to refer to everything that is traced -(sequences, refs, strings). -This section describes how the GC works. - -The basic algorithm is *Deferrent Reference Counting* with cycle detection. -References on the stack are not counted for better performance and easier C -code generation. - -Each cell has a header consisting of a RC and a pointer to its type -descriptor. However the program does not know about these, so they are placed at -negative offsets. In the GC code the type ``PCell`` denotes a pointer -decremented by the right offset, so that the header can be accessed easily. It -is extremely important that ``pointer`` is not confused with a ``PCell`` -as this would lead to a memory corruption. - - -The CellSet data structure --------------------------- - -The GC depends on an extremely efficient datastructure for storing a -set of pointers - this is called a ``TCellSet`` in the source code. -Inserting, deleting and searching are done in constant time. However, -modifying a ``TCellSet`` during traversal leads to undefined behaviour. - -.. code-block:: Nim - type - TCellSet # hidden - - proc cellSetInit(s: var TCellSet) # initialize a new set - proc cellSetDeinit(s: var TCellSet) # empty the set and free its memory - proc incl(s: var TCellSet, elem: PCell) # include an element - proc excl(s: var TCellSet, elem: PCell) # exclude an element - - proc `in`(elem: PCell, s: TCellSet): bool # tests membership - - iterator elements(s: TCellSet): (elem: PCell) - - -All the operations have to perform efficiently. Because a Cellset can -become huge a hash table alone is not suitable for this. - -We use a mixture of bitset and hash table for this. The hash table maps *pages* -to a page descriptor. The page descriptor contains a bit for any possible cell -address within this page. So including a cell is done as follows: - -- Find the page descriptor for the page the cell belongs to. -- Set the appropriate bit in the page descriptor indicating that the - cell points to the start of a memory block. - -Removing a cell is analogous - the bit has to be set to zero. -Single page descriptors are never deleted from the hash table. This is not -needed as the data structures needs to be rebuilt periodically anyway. - -Complete traversal is done in this way:: - - for each page descriptor d: - for each bit in d: - if bit == 1: - traverse the pointer belonging to this bit - - -Further complications ---------------------- - -In Nim the compiler cannot always know if a reference -is stored on the stack or not. This is caused by var parameters. -Consider this example: - -.. code-block:: Nim - proc setRef(r: var ref TNode) = - new(r) - - proc usage = - var - r: ref TNode - setRef(r) # here we should not update the reference counts, because - # r is on the stack - setRef(r.left) # here we should update the refcounts! - -We have to decide at runtime whether the reference is on the stack or not. -The generated code looks roughly like this: - -.. code-block:: C - void setref(TNode** ref) { - unsureAsgnRef(ref, newObj(TNode_TI, sizeof(TNode))) - } - void usage(void) { - setRef(&r) - setRef(&r->left) - } - -Note that for systems with a continuous stack (which most systems have) -the check whether the ref is on the stack is very cheap (only two -comparisons). - - -Code generation for closures -============================ - -Code generation for closures is implemented by `lambda lifting`:idx:. - - -Design ------- - -A ``closure`` proc var can call ordinary procs of the default Nim calling -convention. But not the other way round! A closure is implemented as a -``tuple[prc, env]``. ``env`` can be nil implying a call without a closure. -This means that a call through a closure generates an ``if`` but the -interoperability is worth the cost of the ``if``. Thunk generation would be -possible too, but it's slightly more effort to implement. - -Tests with GCC on Amd64 showed that it's really beneficial if the -'environment' pointer is passed as the last argument, not as the first argument. - -Proper thunk generation is harder because the proc that is to wrap -could stem from a complex expression: - -.. code-block:: nim - receivesClosure(returnsDefaultCC[i]) - -A thunk would need to call 'returnsDefaultCC[i]' somehow and that would require -an *additional* closure generation... Ok, not really, but it requires to pass -the function to call. So we'd end up with 2 indirect calls instead of one. -Another much more severe problem which this solution is that it's not GC-safe -to pass a proc pointer around via a generic ``ref`` type. - - -Example code: - -.. code-block:: nim - proc add(x: int): proc (y: int): int {.closure.} = - return proc (y: int): int = - return x + y - - var add2 = add(2) - echo add2(5) #OUT 7 - -This should produce roughly this code: - -.. code-block:: nim - type - PEnv = ref object - x: int # data - - proc anon(y: int, c: PEnv): int = - return y + c.x - - proc add(x: int): tuple[prc, data] = - var env: PEnv - new env - env.x = x - result = (anon, env) - - var add2 = add(2) - let tmp = if add2.data == nil: add2.prc(5) else: add2.prc(5, add2.data) - echo tmp - - -Beware of nesting: - -.. code-block:: nim - proc add(x: int): proc (y: int): proc (z: int): int {.closure.} {.closure.} = - return lambda (y: int): proc (z: int): int {.closure.} = - return lambda (z: int): int = - return x + y + z - - var add24 = add(2)(4) - echo add24(5) #OUT 11 - -This should produce roughly this code: - -.. code-block:: nim - type - PEnvX = ref object - x: int # data - - PEnvY = ref object - y: int - ex: PEnvX - - proc lambdaZ(z: int, ey: PEnvY): int = - return ey.ex.x + ey.y + z - - proc lambdaY(y: int, ex: PEnvX): tuple[prc, data: PEnvY] = - var ey: PEnvY - new ey - ey.y = y - ey.ex = ex - result = (lambdaZ, ey) - - proc add(x: int): tuple[prc, data: PEnvX] = - var ex: PEnvX - ex.x = x - result = (labmdaY, ex) - - var tmp = add(2) - var tmp2 = tmp.fn(4, tmp.data) - var add24 = tmp2.fn(4, tmp2.data) - echo add24(5) - - -We could get rid of nesting environments by always inlining inner anon procs. -More useful is escape analysis and stack allocation of the environment, -however. - - -Alternative ------------ - -Process the closure of all inner procs in one pass and accumulate the -environments. This is however not always possible. - - -Accumulator ------------ - -.. code-block:: nim - proc getAccumulator(start: int): proc (): int {.closure} = - var i = start - return lambda: int = - inc i - return i - - proc p = - var delta = 7 - proc accumulator(start: int): proc(): int = - var x = start-1 - result = proc (): int = - x = x + delta - inc delta - return x - - var a = accumulator(3) - var b = accumulator(4) - echo a() + b() - - -Internals ---------- - -Lambda lifting is implemented as part of the ``transf`` pass. The ``transf`` -pass generates code to setup the environment and to pass it around. However, -this pass does not change the types! So we have some kind of mismatch here; on -the one hand the proc expression becomes an explicit tuple, on the other hand -the tyProc(ccClosure) type is not changed. For C code generation it's also -important the hidden formal param is ``void*`` and not something more -specialized. However the more specialized env type needs to passed to the -backend somehow. We deal with this by modifying ``s.ast[paramPos]`` to contain -the formal hidden parameter, but not ``s.typ``! - - -Integer literals: ------------------ - -In Nim, there is a redundant way to specify the type of an -integer literal. First of all, it should be unsurprising that every -node has a node kind. The node of an integer literal can be any of the -following values: - - nkIntLit, nkInt8Lit, nkInt16Lit, nkInt32Lit, nkInt64Lit, - nkUIntLit, nkUInt8Lit, nkUInt16Lit, nkUInt32Lit, nkUInt64Lit - -On top of that, there is also the `typ` field for the type. It the -kind of the `typ` field can be one of the following ones, and it -should be matching the literal kind: - - tyInt, tyInt8, tyInt16, tyInt32, tyInt64, tyUInt, tyUInt8, - tyUInt16, tyUInt32, tyUInt64 - -Then there is also the integer literal type. This is a specific type -that is implicitly convertible into the requested type if the -requested type can hold the value. For this to work, the type needs to -know the concrete value of the literal. For example an expression -`321` will be of type `int literal(321)`. This type is implicitly -convertible to all integer types and ranges that contain the value -`321`. That would be all builtin integer types except `uint8` and -`int8` where `321` would be out of range. When this literal type is -assigned to a new `var` or `let` variable, it's type will be resolved -to just `int`, not `int literal(321)` unlike constants. A constant -keeps the full `int literal(321)` type. Here is an example where that -difference matters. - - -.. code-block:: nim - - proc foo(arg: int8) = - echo "def" - - const tmp1 = 123 - foo(tmp1) # OK - - let tmp2 = 123 - foo(tmp2) # Error - -In a context with multiple overloads, the integer literal kind will -always prefer the `int` type over all other types. If none of the -overloads is of type `int`, then there will be an error because of -ambiguity. - -.. code-block:: nim - - proc foo(arg: int) = - echo "abc" - proc foo(arg: int8) = - echo "def" - foo(123) # output: abc - - proc bar(arg: int16) = - echo "abc" - proc bar(arg: int8) = - echo "def" - - bar(123) # Error ambiguous call - -In the compiler these integer literal types are represented with the -node kind `nkIntLit`, type kind `tyInt` and the member `n` of the type -pointing back to the integer literal node in the ast containing the -integer value. These are the properties that hold true for integer -literal types. - - n.kind == nkIntLit - n.typ.kind == tyInt - n.typ.n == n - -Other literal types, such as `uint literal(123)` that would -automatically convert to other integer types, but prefers to -become a `uint` are not part of the Nim language. - -In an unchecked AST, the `typ` field is nil. The type checker will set -the `typ` field accordingly to the node kind. Nodes of kind `nkIntLit` -will get the integer literal type (e.g. `int literal(123)`). Nodes of -kind `nkUIntLit` will get type `uint` (kind `tyUint`), etc. - -This also means that it is not possible to write a literal in an -unchecked AST that will after sem checking just be of type `int` and -not implicitly convertible to other integer types. This only works for -all integer types that are not `int`. diff --git a/doc/koch.md b/doc/koch.md new file mode 100644 index 000000000..8fa19ce44 --- /dev/null +++ b/doc/koch.md @@ -0,0 +1,91 @@ +=============================== + Nim maintenance script +=============================== + +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +> "A great chef is an artist that I truly respect" -- Robert Stack. + + +Introduction +============ + +The `koch`:idx: program is Nim's maintenance script. It is a replacement +for make and shell scripting with the advantage that it is much more portable. +The word *koch* means *cook* in German. `koch`:cmd: is used mainly to build the +Nim compiler, but it can also be used for other tasks. This document +describes the supported commands and their options. + + +Commands +======== + +boot command +------------ + +The `boot`:idx: command bootstraps the compiler, and it accepts different +options: + +-d:release By default a debug version is created, passing this option will + force a release build, which is much faster and should be preferred + unless you are debugging the compiler. +-d:nimUseLinenoise Use the linenoise library for interactive mode + (not needed on Windows). +-d:leanCompiler Produce a compiler without JS codegen or + documentation generator in order to use less RAM + for bootstrapping. + +After compilation is finished you will hopefully end up with the nim +compiler in the `bin` directory. You can add Nim's `bin` directory to +your `$PATH` or use the install command to place it where it will be +found. + +csource command +--------------- + +The `csource`:idx: command builds the C sources for installation. It accepts +the same options as you would pass to the [boot command]( +#commands-boot-command). + +temp command +------------ + +The temp command builds the Nim compiler but with a different final name +(`nim_temp`:cmd:), so it doesn't overwrite your normal compiler. You can use +this command to test different options, the same you would issue for the [boot +command](#commands-boot-command). + +test command +------------ + +The `test`:idx: command can also be invoked with the alias `tests`:option:. This +command will compile and run ``testament/tester.nim``, which is the main +driver of Nim's test suite. You can pass options to the `test`:option: command, +they will be forwarded to the tester. See its source code for available +options. + +web command +----------- + +The `web`:idx: command converts the documentation in the `doc` directory +from rst to HTML. It also repeats the same operation but places the result in +the ``web/upload`` which can be used to update the website at +https://nim-lang.org. + +By default, the documentation will be built in parallel using the number of +available CPU cores. If any documentation build sub-commands fail, they will +be rerun in serial fashion so that meaningful error output can be gathered for +inspection. The `--parallelBuild:n`:option: switch or configuration option can be +used to force a specific number of parallel jobs or run everything serially +from the start (`n == 1`). + +pdf command +----------- + +The `pdf`:idx: command builds PDF versions of Nim documentation: Manual, +Tutorial and a few other documents. To run it one needs to +[install Latex/xelatex](https://www.latex-project.org/get) first. diff --git a/doc/koch.rst b/doc/koch.rst deleted file mode 100644 index e984a71ea..000000000 --- a/doc/koch.rst +++ /dev/null @@ -1,83 +0,0 @@ -=============================== - Nim maintenance script -=============================== - -:Version: |nimversion| - -.. contents:: - -.. raw:: html - <blockquote><p> - "A great chef is an artist that I truly respect" -- Robert Stack. - </p></blockquote> - - -Introduction -============ - -The `koch`:idx: program is Nim's maintenance script. It is a replacement -for make and shell scripting with the advantage that it is much more portable. -The word *koch* means *cook* in German. ``koch`` is used mainly to build the -Nim compiler, but it can also be used for other tasks. This document -describes the supported commands and their options. - - -Commands -======== - -boot command ------------- - -The `boot`:idx: command bootstraps the compiler, and it accepts different -options: - --d:release - By default a debug version is created, passing this option will - force a release build, which is much faster and should be preferred - unless you are debugging the compiler. --d:nimUseLinenoise - Use the linenoise library for interactive mode (not needed on Windows). - -After compilation is finished you will hopefully end up with the nim -compiler in the ``bin`` directory. You can add Nim's ``bin`` directory to -your ``$PATH`` or use the install command to place it where it will be -found. - -csource command ---------------- - -The `csource`:idx: command builds the C sources for installation. It accepts -the same options as you would pass to the `boot command -<#commands-boot-command>`_. - -temp command ------------- - -The temp command builds the Nim compiler but with a different final name -(``nim_temp``), so it doesn't overwrite your normal compiler. You can use -this command to test different options, the same you would issue for the `boot -command <#commands-boot-command>`_. - -test command ------------- - -The `test`:idx: command can also be invoked with the alias ``tests``. This -command will compile and run ``testament/tester.nim``, which is the main -driver of Nim's test suite. You can pass options to the ``test`` command, -they will be forwarded to the tester. See its source code for available -options. - -web command ------------ - -The `web`:idx: command converts the documentation in the ``doc`` directory -from rst to HTML. It also repeats the same operation but places the result in -the ``web/upload`` which can be used to update the website at -https://nim-lang.org. - -By default the documentation will be built in parallel using the number of -available CPU cores. If any documentation build sub commands fail, they will -be rerun in serial fashion so that meaningful error output can be gathered for -inspection. The ``--parallelBuild:n`` switch or configuration option can be -used to force a specific number of parallel jobs or run everything serially -from the start (``n == 1``). diff --git a/doc/lib.md b/doc/lib.md new file mode 100644 index 000000000..1507bbaac --- /dev/null +++ b/doc/lib.md @@ -0,0 +1,682 @@ +==================== +Nim Standard Library +==================== + +:Author: Andreas Rumpf +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +Nim's library is divided into *pure libraries*, *impure libraries*, and *wrappers*. + +Pure libraries do not depend on any external ``*.dll`` or ``lib*.so`` binary +while impure libraries do. A wrapper is an impure library that is a very +low-level interface to a C library. + +Read [this document](apis.html) for a quick overview of the API design. + + +Nimble +====== + +Nim's standard library only covers the basics, check +out https://nimble.directory/ for a list of 3rd party packages. + + +Pure libraries +============== + +Automatic imports +----------------- + +* [system](system.html) + Basic procs and operators that every program needs. It also provides IO + facilities for reading and writing text and binary files. It is imported + implicitly by the compiler. Do not import it directly. It relies on compiler + magic to work. + +Core +---- + +* [atomics](atomics.html) + Types and operations for atomic operations and lockless algorithms. + +* [bitops](bitops.html) + Provides a series of low-level methods for bit manipulation. + +* [compilesettings](compilesettings.html) + Querying the compiler about diverse configuration settings from code. + +* [cpuinfo](cpuinfo.html) + Procs to determine the number of CPUs / cores. + +* [effecttraits](effecttraits.html) + Access to the inferred .raises effects + for Nim's macro system. + +* [endians](endians.html) + Helpers that deal with different byte orders. + +* [locks](locks.html) + Locks and condition variables for Nim. + +* [macrocache](macrocache.html) + Provides an API for macros to collect compile-time information across modules. + +* [macros](macros.html) + Contains the AST API and documentation of Nim for writing macros. + +* [rlocks](rlocks.html) + Reentrant locks for Nim. + +* [typeinfo](typeinfo.html) + Provides (unsafe) access to Nim's run-time type information. + +* [typetraits](typetraits.html) + Compile-time reflection procs for working with types. + +* [volatile](volatile.html) + Code for generating volatile loads and stores, + which are useful in embedded and systems programming. + + +Algorithms +---------- + +* [algorithm](algorithm.html) + Some common generic algorithms like sort or binary search. + +* [enumutils](enumutils.html) + Additional functionality for the built-in `enum` type. + +* [sequtils](sequtils.html) + Operations for the built-in `seq` type + which were inspired by functional programming languages. + +* [setutils](setutils.html) + Additional functionality for the built-in `set` type. + + +Collections +----------- + +* [critbits](critbits.html) + A *crit bit tree* which is an efficient + container for a sorted set of strings, or a sorted mapping of strings. + +* [deques](deques.html) + Implementation of a double-ended queue. + The underlying implementation uses a `seq`. + +* [heapqueue](heapqueue.html) + Implementation of a binary heap data structure that can be used as a priority queue. + +* [intsets](intsets.html) + Efficient implementation of a set of ints as a sparse bit set. + +* [lists](lists.html) + Nim linked list support. Contains singly and doubly linked lists and + circular lists ("rings"). + +* [options](options.html) + The option type encapsulates an optional value. + +* [packedsets](packedsets.html) + Efficient implementation of a set of ordinals as a sparse bit set. + +* [ropes](ropes.html) + A *rope* data type. + Ropes can represent very long strings efficiently; + in particular, concatenation is done in O(1) instead of O(n). + +* [sets](sets.html) + Nim hash set support. + +* [strtabs](strtabs.html) + The `strtabs` module implements an efficient hash table that is a mapping + from strings to strings. Supports a case-sensitive, case-insensitive and + style-insensitive modes. + +* [tables](tables.html) + Nim hash table support. Contains tables, ordered tables, and count tables. + + +String handling +--------------- + +* [cstrutils](cstrutils.html) + Utilities for `cstring` handling. + +* [editdistance](editdistance.html) + An algorithm to compute the edit distance between two + Unicode strings. + +* [encodings](encodings.html) + Converts between different character encodings. On UNIX, this uses + the `iconv` library, on Windows the Windows API. + +* [formatfloat](formatfloat.html) + Formatting floats as strings. + +* [objectdollar](objectdollar.html) + A generic `$` operator to convert objects to strings. + +* [punycode](punycode.html) + Implements a representation of Unicode with the limited ASCII character subset. + +* [strbasics](strbasics.html) + Some high performance string operations. + +* [strformat](strformat.html) + Macro based standard string interpolation/formatting. Inspired by + Python's f-strings.\ + **Note:** if you need templating, consider using Nim + [Source Code Filters (SCF)](filters.html). + +* [strmisc](strmisc.html) + Uncommon string handling operations that do not + fit with the commonly used operations in [strutils](strutils.html). + +* [strscans](strscans.html) + A `scanf` macro for convenient parsing of mini languages. + +* [strutils](strutils.html) + Common string handling operations like changing + case of a string, splitting a string into substrings, searching for + substrings, replacing substrings. + +* [unicode](unicode.html) + Support for handling the Unicode UTF-8 encoding. + +* [unidecode](unidecode.html) + It provides a single proc that does Unicode to ASCII transliterations. + Based on Python's Unidecode module. + +* [widestrs](widestrs.html) + Nim support for C/C++'s wide strings. + +* [wordwrap](wordwrap.html) + An algorithm for word-wrapping Unicode strings. + + +Time handling +------------- + +* [monotimes](monotimes.html) + The `monotimes` module implements monotonic timestamps. + +* [times](times.html) + The `times` module contains support for working with time. + + +Generic Operating System Services +--------------------------------- + +* [appdirs](appdirs.html) + Helpers for determining special directories used by apps. + +* [cmdline](cmdline.html) + System facilities for reading command line parameters. + +* [dirs](dirs.html) + Directory handling. + +* [distros](distros.html) + Basics for OS distribution ("distro") detection + and the OS's native package manager. + Its primary purpose is to produce output for Nimble packages, + but it also contains the widely used **Distribution** enum + that is useful for writing platform-specific code. + See [packaging](packaging.html) for hints on distributing Nim using OS packages. + +* [dynlib](dynlib.html) + Accessing symbols from shared libraries. + +* [envvars](envvars.html) + Environment variable handling. + +* [exitprocs](exitprocs.html) + Adding hooks to program exit. + +* [files](files.html) + File handling. + +* [memfiles](memfiles.html) + Support for memory-mapped files (Posix's `mmap`) + on the different operating systems. + +* [os](os.html) + Basic operating system facilities like retrieving environment variables, + reading command line arguments, working with directories, running shell + commands, etc. + +* [oserrors](oserrors.html) + OS error reporting. + +* [osproc](osproc.html) + Module for process communication beyond `os.execShellCmd`. + +* [paths](paths.html) + Path handling. + +* [reservedmem](reservedmem.html) + Utilities for reserving portions of the + address space of a program without consuming physical memory. + +* [streams](streams.html) + A stream interface and two implementations thereof: + the `FileStream` and the `StringStream` which implement the stream + interface for Nim file objects (`File`) and strings. Other modules + may provide other implementations for this standard stream interface. + +* [symlinks](symlinks.html) + Symlink handling. + +* [syncio](syncio.html) + Various synchronized I/O operations. + +* [terminal](terminal.html) + A module to control the terminal output (also called *console*). + +* [tempfiles](tempfiles.html) + Some utilities for generating temporary path names and + creating temporary files and directories. + + +Math libraries +-------------- + +* [complex](complex.html) + Complex numbers and relevant mathematical operations. + +* [fenv](fenv.html) + Floating-point environment. Handling of floating-point rounding and + exceptions (overflow, zero-divide, etc.). + +* [lenientops](lenientops.html) + Binary operators for mixed integer/float expressions for convenience. + +* [math](math.html) + Mathematical operations like cosine, square root. + +* [random](random.html) + Fast and tiny random number generator. + +* [rationals](rationals.html) + Rational numbers and relevant mathematical operations. + +* [stats](stats.html) + Statistical analysis. + +* [sysrand](sysrand.html) + Cryptographically secure pseudorandom number generator. + + +Internet Protocols and Support +------------------------------ + +* [async](async.html) + Exports `asyncmacro` and `asyncfutures` for native backends, and `asyncjs` on the JS backend. + +* [asyncdispatch](asyncdispatch.html) + An asynchronous dispatcher for IO operations. + +* [asyncfile](asyncfile.html) + An asynchronous file reading and writing using `asyncdispatch`. + +* [asyncftpclient](asyncftpclient.html) + An asynchronous FTP client using the `asyncnet` module. + +* [asynchttpserver](asynchttpserver.html) + An asynchronous HTTP server using the `asyncnet` module. + +* [asyncmacro](asyncmacro.html) + `async` and `multisync` macros for `asyncdispatch`. + +* [asyncnet](asyncnet.html) + Asynchronous sockets based on the `asyncdispatch` module. + +* [asyncstreams](asyncstreams.html) + `FutureStream` - a future that acts as a queue. + +* [cgi](cgi.html) + Helpers for CGI applications. + +* [cookies](cookies.html) + Helper procs for parsing and generating cookies. + +* [httpclient](httpclient.html) + A simple HTTP client with support for both synchronous + and asynchronous retrieval of web pages. + +* [mimetypes](mimetypes.html) + A mimetypes database. + +* [nativesockets](nativesockets.html) + A low-level sockets API. + +* [net](net.html) + A high-level sockets API. + +* [selectors](selectors.html) + A selector API with backends specific to each OS. + Supported OS primitives: `epoll`, `kqueue`, `poll`, and `select` on Windows. + +* [smtp](smtp.html) + A simple SMTP client with support for both synchronous and asynchronous operation. + +* [socketstreams](socketstreams.html) + An implementation of the streams interface for sockets. + +* [uri](uri.html) + Functions for working with URIs and URLs. + + +Threading +--------- + +* [isolation](isolation.html) + The `Isolated[T]` type for + safe construction of isolated subgraphs that can be + passed efficiently to different channels and threads. + +* [tasks](tasks.html) + Basic primitives for creating parallel programs. + +* [threadpool](threadpool.html) + Implements Nim's [spawn](manual_experimental.html#parallel-amp-spawn). + +* [typedthreads](typedthreads.html) + Basic Nim thread support. + + +Parsers +------- + +* [htmlparser](htmlparser.html) + HTML document parser that creates a XML tree representation. + +* [json](json.html) + High-performance JSON parser. + +* [lexbase](lexbase.html) + A low-level module that implements an extremely efficient buffering + scheme for lexers and parsers. This is used by the diverse parsing modules. + +* [parsecfg](parsecfg.html) + The `parsecfg` module implements a high-performance configuration file + parser. The configuration file's syntax is similar to the Windows ``.ini`` + format, but much more powerful, as it is not a line based parser. String + literals, raw string literals, and triple quote string literals are supported + as in the Nim programming language. + +* [parsecsv](parsecsv.html) + The `parsecsv` module implements a simple high-performance CSV parser. + +* [parsejson](parsejson.html) + A JSON parser. It is used and exported by the [json](json.html) module, but can also be used in its own right. + +* [parseopt](parseopt.html) + The `parseopt` module implements a command line option parser. + +* [parsesql](parsesql.html) + The `parsesql` module implements a simple high-performance SQL parser. + +* [parseutils](parseutils.html) + Helpers for parsing tokens, numbers, identifiers, etc. + +* [parsexml](parsexml.html) + The `parsexml` module implements a simple high performance XML/HTML parser. + The only encoding that is supported is UTF-8. The parser has been designed + to be somewhat error-correcting, so that even some "wild HTML" found on the + web can be parsed with it. + +* [pegs](pegs.html) + Procedures and operators for handling PEGs. + + +Docutils +-------- + +* [packages/docutils/highlite](highlite.html) + Source highlighter for programming or markup languages. Currently, + only a few languages are supported, other languages may be added. + The interface supports one language nested in another. + +* [packages/docutils/rst](rst.html) + A reStructuredText parser. A large subset + is implemented. Some features of the markdown wiki syntax are also supported. + +* [packages/docutils/rstast](rstast.html) + An AST for the reStructuredText parser. + +* [packages/docutils/rstgen](rstgen.html) + A generator of HTML/Latex from reStructuredText. + + +XML Processing +-------------- + +* [xmltree](xmltree.html) + A simple XML tree. More efficient and simpler than the DOM. It also + contains a macro for XML/HTML code generation. + +* [xmlparser](xmlparser.html) + XML document parser that creates a XML tree representation. + + +Generators +---------- + +* [genasts](genasts.html) + AST generation using captured variables for macros. + +* [htmlgen](htmlgen.html) + A simple XML and HTML code + generator. Each commonly used HTML tag has a corresponding macro + that generates a string with its HTML representation. + + +Hashing +------- + +* [base64](base64.html) + A Base64 encoder and decoder. + +* [hashes](hashes.html) + Efficient computations of hash values for diverse Nim types. + +* [md5](md5.html) + The MD5 checksum algorithm. + +* [oids](oids.html) + An OID is a global ID that consists of a timestamp, + a unique counter, and a random value. This combination should suffice to + produce a globally distributed unique ID. + +* [sha1](sha1.html) + The SHA-1 checksum algorithm. + + +Serialization +------------- + +* [jsonutils](jsonutils.html) + Hookable (de)serialization for arbitrary types + using JSON. + +* [marshal](marshal.html) + Contains procs for serialization and deserialization of arbitrary Nim + data structures. + + +Miscellaneous +------------- + +* [assertions](assertions.html) + Assertion handling. + +* [browsers](browsers.html) + Procs for opening URLs with the user's default + browser. + +* [colors](colors.html) + Color handling. + +* [coro](coro.html) + Experimental coroutines in Nim. + +* [decls](decls.html) + Syntax sugar for some declarations. + +* [enumerate](enumerate.html) + `enumerate` syntactic sugar based on Nim's macro system. + +* [importutils](importutils.html) + Utilities related to import and symbol resolution. + +* [logging](logging.html) + A simple logger. + +* [segfaults](segfaults.html) + Turns access violations or segfaults into a `NilAccessDefect` exception. + +* [sugar](sugar.html) + Nice syntactic sugar based on Nim's macro system. + +* [unittest](unittest.html) + Implements a Unit testing DSL. + +* [varints](varints.html) + Decode variable-length integers that are compatible with SQLite. + +* [with](with.html) + The `with` macro for easy function chaining. + +* [wrapnils](wrapnils.html) + Allows evaluating expressions safely against nil dereferences. + + +Modules for the JavaScript backend +---------------------------------- + +* [asyncjs](asyncjs.html) + Types and macros for writing asynchronous procedures in JavaScript. + +* [dom](dom.html) + Declaration of the Document Object Model for the JS backend. + +* [jsbigints](jsbigints.html) + Arbitrary precision integers. + +* [jsconsole](jsconsole.html) + Wrapper for the `console` object. + +* [jscore](jscore.html) + The wrapper of core JavaScript functions. For most purposes, you should be using + the `math`, `json`, and `times` stdlib modules instead of this module. + +* [jsfetch](jsfetch.html) + Wrapper for `fetch`. + +* [jsffi](jsffi.html) + Types and macros for easier interaction with JavaScript. + +* [jsre](jsre.html) + Regular Expressions for the JavaScript target. + + +Impure libraries +================ + +Regular expressions +------------------- + +* [re](re.html) + Procedures and operators for handling regular + expressions. The current implementation uses PCRE. + +* [nre](nre.html) + + Many help functions for handling regular expressions. + The current implementation uses PCRE. + +Database support +---------------- + +* [db_mysql](db_mysql.html) + A higher level MySQL database wrapper. The same interface is implemented + for other databases too. + +* [db_odbc](db_odbc.html) + A higher level ODBC database wrapper. The same interface is implemented + for other databases too. + +* [db_postgres](db_postgres.html) + A higher level PostgreSQL database wrapper. The same interface is implemented + for other databases too. + +* [db_sqlite](db_sqlite.html) + A higher level SQLite database wrapper. The same interface is implemented + for other databases too. + + +Generic Operating System Services +--------------------------------- + +* [rdstdin](rdstdin.html) + Code for reading user input from stdin. + + +Wrappers +======== + +The generated HTML for some of these wrappers is so huge that it is +not contained in the distribution. You can then find them on the website. + + +Windows-specific +---------------- + +* [winlean](winlean.html) + Wrapper for a small subset of the Win32 API. +* [registry](registry.html) + Windows registry support. + + +UNIX specific +------------- + +* [posix](posix.html) + Wrapper for the POSIX standard. +* [posix_utils](posix_utils.html) + Contains helpers for the POSIX standard or specialized for Linux and BSDs. + + +Regular expressions +------------------- + +* [pcre](pcre.html) + Wrapper for the PCRE library. + + +Database support +---------------- + +* [mysql](mysql.html) + Wrapper for the mySQL API. +* [odbcsql](odbcsql.html) + interface to the ODBC driver. +* [postgres](postgres.html) + Wrapper for the PostgreSQL API. +* [sqlite3](sqlite3.html) + Wrapper for the SQLite 3 API. + + +Network Programming and Internet Protocols +------------------------------------------ + +* [openssl](openssl.html) + Wrapper for OpenSSL. diff --git a/doc/lib.rst b/doc/lib.rst deleted file mode 100644 index e05657b42..000000000 --- a/doc/lib.rst +++ /dev/null @@ -1,568 +0,0 @@ -==================== -Nim Standard Library -==================== - -:Author: Andreas Rumpf -:Version: |nimversion| - -.. contents:: - -Nim's library is divided into *pure libraries*, *impure libraries* and *wrappers*. - -Pure libraries do not depend on any external ``*.dll`` or ``lib*.so`` binary -while impure libraries do. A wrapper is an impure library that is a very -low-level interface to a C library. - -Read this `document <apis.html>`_ for a quick overview of the API design. - - -Nimble -====== - -Nim's standard library only covers the basics, check -out `<https://nimble.directory/>`_ for a list of 3rd party packages. - - -Pure libraries -============== - -Automatic imports ------------------ - -* `system <system.html>`_ - Basic procs and operators that every program needs. It also provides IO - facilities for reading and writing text and binary files. It is imported - implicitly by the compiler. Do not import it directly. It relies on compiler - magic to work. - -* `threads <threads.html>`_ - Basic Nim thread support. **Note**: This is part of the system module. Do not - import it explicitly. Enabled with ``--threads:on``. - -* `channels <channels.html>`_ - Nim message passing support for threads. **Note**: This is part of the - system module. Do not import it explicitly. Enabled with ``--threads:on``. - - -Core ----- - -* `bitops <bitops.html>`_ - Provides a series of low level methods for bit manipulation. - -* `cpuinfo <cpuinfo.html>`_ - This module implements procs to determine the number of CPUs / cores. - -* `endians <endians.html>`_ - This module contains helpers that deal with different byte orders. - -* `lenientops <lenientops.html>`_ - Provides binary operators for mixed integer/float expressions for convenience. - -* `locks <locks.html>`_ - Locks and condition variables for Nim. - -* `macros <macros.html>`_ - Contains the AST API and documentation of Nim for writing macros. - -* `rlocks <rlocks.html>`_ - Reentrant locks for Nim. - -* `typeinfo <typeinfo.html>`_ - Provides (unsafe) access to Nim's run time type information. - -* `typetraits <typetraits.html>`_ - This module defines compile-time reflection procs for working with types. - -* `volatile <volatile.html>`_ - This module contains code for generating volatile loads and stores, - which are useful in embedded and systems programming. - - -Algorithms ----------- - -* `algorithm <algorithm.html>`_ - Implements some common generic algorithms like sort or binary search. - -* `sequtils <sequtils.html>`_ - This module implements operations for the built-in seq type - which were inspired by functional programming languages. - - - -Collections ------------ - -* `critbits <critbits.html>`_ - This module implements a *crit bit tree* which is an efficient - container for a sorted set of strings, or for a sorted mapping of strings. - -* `deques <deques.html>`_ - Implementation of a double-ended queue. - The underlying implementation uses a ``seq``. - -* `heapqueue <heapqueue.html>`_ - Implementation of a heap data structure that can be used as a priority queue. - -* `intsets <intsets.html>`_ - Efficient implementation of a set of ints as a sparse bit set. - -* `lists <lists.html>`_ - Nim linked list support. Contains singly and doubly linked lists and - circular lists ("rings"). - -* `options <options.html>`_ - The option type encapsulates an optional value. - -* `sets <sets.html>`_ - Nim hash and bit set support. - -* `sharedlist <sharedlist.html>`_ - Nim shared linked list support. Contains shared singly linked list. - -* `sharedtables <sharedtables.html>`_ - Nim shared hash table support. Contains shared tables. - -* `tables <tables.html>`_ - Nim hash table support. Contains tables, ordered tables and count tables. - - - -String handling ---------------- - -* `cstrutils <cstrutils.html>`_ - Utilities for ``cstring`` handling. - -* `std/editdistance <editdistance.html>`_ - This module contains an algorithm to compute the edit distance between two - Unicode strings. - -* `encodings <encodings.html>`_ - Converts between different character encodings. On UNIX, this uses - the ``iconv`` library, on Windows the Windows API. - -* `parseutils <parseutils.html>`_ - This module contains helpers for parsing tokens, numbers, identifiers, etc. - -* `pegs <pegs.html>`_ - This module contains procedures and operators for handling PEGs. - -* `punycode <punycode.html>`_ - Implements a representation of Unicode with the limited ASCII character subset. - -* `ropes <ropes.html>`_ - This module contains support for a *rope* data type. - Ropes can represent very long strings efficiently; especially concatenation - is done in O(1) instead of O(n). - -* `strformat <strformat.html>`_ - Macro based standard string interpolation / formatting. Inspired by - Python's ``f``-strings. - -* `strmisc <strmisc.html>`_ - This module contains uncommon string handling operations that do not - fit with the commonly used operations in strutils. - -* `strscans <strscans.html>`_ - This module contains a ``scanf`` macro for convenient parsing of mini languages. - -* `strtabs <strtabs.html>`_ - The ``strtabs`` module implements an efficient hash table that is a mapping - from strings to strings. Supports a case-sensitive, case-insensitive and - style-insensitive modes. - -* `strutils <strutils.html>`_ - This module contains common string handling operations like changing - case of a string, splitting a string into substrings, searching for - substrings, replacing substrings. - -* `unicode <unicode.html>`_ - This module provides support to handle the Unicode UTF-8 encoding. - -* `unidecode <unidecode.html>`_ - It provides a single proc that does Unicode to ASCII transliterations. - Based on Python's Unidecode module. - -* `std/wordwrap <wordwrap.html>`_ - This module contains an algorithm to wordwrap a Unicode string. - - -Time handling -------------- - -* `std/monotimes <monotimes.html>`_ - The `monotimes` module implements monotonic timestamps. - -* `times <times.html>`_ - The ``times`` module contains support for working with time. - - -Generic Operating System Services ---------------------------------- - -* `distros <distros.html>`_ - This module implements the basics for OS distribution ("distro") detection - and the OS's native package manager. - Its primary purpose is to produce output for Nimble packages, - but it also contains the widely used **Distribution** enum - that is useful for writing platform specific code. - See `packaging <packaging.html>`_ for hints on distributing Nim using OS packages. - -* `dynlib <dynlib.html>`_ - This module implements the ability to access symbols from shared libraries. - -* `marshal <marshal.html>`_ - Contains procs for serialization and deserialization of arbitrary Nim - data structures. - -* `memfiles <memfiles.html>`_ - This module provides support for memory mapped files (Posix's ``mmap``) - on the different operating systems. - -* `os <os.html>`_ - Basic operating system facilities like retrieving environment variables, - reading command line arguments, working with directories, running shell - commands, etc. - -* `osproc <osproc.html>`_ - Module for process communication beyond ``os.execShellCmd``. - -* `streams <streams.html>`_ - This module provides a stream interface and two implementations thereof: - the `FileStream` and the `StringStream` which implement the stream - interface for Nim file objects (`File`) and strings. Other modules - may provide other implementations for this standard stream interface. - -* `terminal <terminal.html>`_ - This module contains a few procedures to control the *terminal* - (also called *console*). The implementation simply uses ANSI escape - sequences and does not depend on any other module. - - -Math libraries --------------- - -* `complex <complex.html>`_ - This module implements complex numbers and their mathematical operations. - -* `fenv <fenv.html>`_ - Floating-point environment. Handling of floating-point rounding and - exceptions (overflow, zero-divide, etc.). - -* `math <math.html>`_ - Mathematical operations like cosine, square root. - -* `mersenne <mersenne.html>`_ - Mersenne twister random number generator. - -* `random <random.html>`_ - Fast and tiny random number generator. - -* `rationals <rationals.html>`_ - This module implements rational numbers and their mathematical operations. - -* `stats <stats.html>`_ - Statistical analysis - -* `std/sums <sums.html>`_ - Fast summation functions. - - -Internet Protocols and Support ------------------------------- - -* `asyncdispatch <asyncdispatch.html>`_ - This module implements an asynchronous dispatcher for IO operations. - -* `asyncfile <asyncfile.html>`_ - This module implements asynchronous file reading and writing using - ``asyncdispatch``. - -* `asyncftpclient <asyncftpclient.html>`_ - This module implements an asynchronous FTP client using the ``asyncnet`` - module. - -* `asynchttpserver <asynchttpserver.html>`_ - This module implements an asynchronous HTTP server using the ``asyncnet`` - module. - -* `asyncnet <asyncnet.html>`_ - This module implements asynchronous sockets based on the ``asyncdispatch`` - module. - -* `asyncstreams <asyncstreams.html>`_ - This module provides `FutureStream` - a future that acts as a queue. - -* `cgi <cgi.html>`_ - This module implements helpers for CGI applications. - -* `cookies <cookies.html>`_ - This module contains helper procs for parsing and generating cookies. - -* `httpclient <httpclient.html>`_ - This module implements a simple HTTP client which supports both synchronous - and asynchronous retrieval of web pages. - -* `mimetypes <mimetypes.html>`_ - This module implements a mimetypes database. - -* `nativesockets <nativesockets.html>`_ - This module implements a low-level sockets API. - -* `net <net.html>`_ - This module implements a high-level sockets API. It replaces the - ``sockets`` module. - -* `selectors <selectors.html>`_ - This module implements a selector API with backends specific to each OS. - Currently epoll on Linux and select on other operating systems. - -* `smtp <smtp.html>`_ - This module implement a simple SMTP client. - -* `uri <uri.html>`_ - This module provides functions for working with URIs. - - -Threading ---------- - -* `threadpool <threadpool.html>`_ - Implements Nim's `spawn <manual_experimental.html#parallel-amp-spawn>`_. - - -Parsers -------- - -* `htmlparser <htmlparser.html>`_ - This module parses an HTML document and creates its XML tree representation. - -* `json <json.html>`_ - High performance JSON parser. - -* `lexbase <lexbase.html>`_ - This is a low level module that implements an extremely efficient buffering - scheme for lexers and parsers. This is used by the diverse parsing modules. - -* `parsecfg <parsecfg.html>`_ - The ``parsecfg`` module implements a high performance configuration file - parser. The configuration file's syntax is similar to the Windows ``.ini`` - format, but much more powerful, as it is not a line based parser. String - literals, raw string literals and triple quote string literals are supported - as in the Nim programming language. - -* `parsecsv <parsecsv.html>`_ - The ``parsecsv`` module implements a simple high performance CSV parser. - -* `parseopt <parseopt.html>`_ - The ``parseopt`` module implements a command line option parser. - -* `parsesql <parsesql.html>`_ - The ``parsesql`` module implements a simple high performance SQL parser. - -* `parsexml <parsexml.html>`_ - The ``parsexml`` module implements a simple high performance XML/HTML parser. - The only encoding that is supported is UTF-8. The parser has been designed - to be somewhat error correcting, so that even some "wild HTML" found on the - Web can be parsed with it. - - -Docutils --------- - -* `packages/docutils/highlite <highlite.html>`_ - Source highlighter for programming or markup languages. Currently - only few languages are supported, other languages may be added. - The interface supports one language nested in another. - -* `packages/docutils/rst <rst.html>`_ - This module implements a reStructuredText parser. A large subset - is implemented. Some features of the markdown wiki syntax are - also supported. - -* `packages/docutils/rstast <rstast.html>`_ - This module implements an AST for the reStructuredText parser. - -* `packages/docutils/rstgen <rstgen.html>`_ - This module implements a generator of HTML/Latex from reStructuredText. - - -XML Processing --------------- - -* `xmltree <xmltree.html>`_ - A simple XML tree. More efficient and simpler than the DOM. It also - contains a macro for XML/HTML code generation. - -* `xmlparser <xmlparser.html>`_ - This module parses an XML document and creates its XML tree representation. - - -Generators ----------- - -* `htmlgen <htmlgen.html>`_ - This module implements a simple XML and HTML code - generator. Each commonly used HTML tag has a corresponding macro - that generates a string with its HTML representation. - - - -Hashing -------- - -* `base64 <base64.html>`_ - This module implements a base64 encoder and decoder. - -* `hashes <hashes.html>`_ - This module implements efficient computations of hash values for diverse - Nim types. - -* `md5 <md5.html>`_ - This module implements the MD5 checksum algorithm. - -* `oids <oids.html>`_ - An OID is a global ID that consists of a timestamp, - a unique counter and a random value. This combination should suffice to - produce a globally distributed unique ID. This implementation was extracted - from the Mongodb interface and it thus binary compatible with a Mongo OID. - -* `std/sha1 <sha1.html>`_ - This module implements a sha1 encoder and decoder. - - - -Miscellaneous -------------- - -* `browsers <browsers.html>`_ - This module implements procs for opening URLs with the user's default - browser. - -* `colors <colors.html>`_ - This module implements color handling for Nim. - -* `coro <coro.html>`_ - This module implements experimental coroutines in Nim. - -* `logging <logging.html>`_ - This module implements a simple logger. - -* `segfaults <segfaults.html>`_ - Turns access violations or segfaults into a ``NilAccessDefect`` exception. - -* `sugar <sugar.html>`_ - This module implements nice syntactic sugar based on Nim's macro system. - -* `unittest <unittest.html>`_ - Implements a Unit testing DSL. - -* `std/varints <varints.html>`_ - Decode variable length integers that are compatible with SQLite. - - -Modules for JS backend ----------------------- - -* `asyncjs <asyncjs.html>`_ - Types and macros for writing asynchronous procedures in JavaScript. - -* `dom <dom.html>`_ - Declaration of the Document Object Model for the JS backend. - -* `jsconsole <jsconsole.html>`_ - Wrapper for the ``console`` object. - -* `jscore <jscore.html>`_ - Wrapper of core JavaScript functions. For most purposes you should be using - the ``math``, ``json``, and ``times`` stdlib modules instead of this module. - -* `jsffi <jsffi.html>`_ - Types and macros for easier interaction with JavaScript. - - -Impure libraries -================ - -Regular expressions -------------------- - -* `re <re.html>`_ - This module contains procedures and operators for handling regular - expressions. The current implementation uses PCRE. - - -Database support ----------------- - -* `db_postgres <db_postgres.html>`_ - A higher level PostgreSQL database wrapper. The same interface is implemented - for other databases too. - -* `db_mysql <db_mysql.html>`_ - A higher level MySQL database wrapper. The same interface is implemented - for other databases too. - -* `db_sqlite <db_sqlite.html>`_ - A higher level SQLite database wrapper. The same interface is implemented - for other databases too. - - -Wrappers -======== - -The generated HTML for some of these wrappers is so huge that it is -not contained in the distribution. You can then find them on the website. - - -Windows specific ----------------- - -* `winlean <winlean.html>`_ - Contains a wrapper for a small subset of the Win32 API. -* `registry <registry.html>`_ - Windows registry support. - - -UNIX specific -------------- - -* `posix <posix.html>`_ - Contains a wrapper for the POSIX standard. -* `posix_utils <posix_utils.html>`_ - Contains helpers for the POSIX standard or specialized for Linux and BSDs. - -Regular expressions -------------------- - -* `pcre <pcre.html>`_ - Wrapper for the PCRE library. - - -GUI libraries -------------- - -* `iup <iup.html>`_ - Wrapper of the IUP GUI library. - - -Database support ----------------- - -* `postgres <postgres.html>`_ - Contains a wrapper for the PostgreSQL API. -* `mysql <mysql.html>`_ - Contains a wrapper for the mySQL API. -* `sqlite3 <sqlite3.html>`_ - Contains a wrapper for SQLite 3 API. -* `odbcsql <odbcsql.html>`_ - interface to the ODBC driver. - - -Network Programming and Internet Protocols ------------------------------------------- - -* `openssl <openssl.html>`_ - Wrapper for OpenSSL. diff --git a/doc/manual.rst b/doc/manual.md index 99888ad53..5c36a0a7b 100644 --- a/doc/manual.rst +++ b/doc/manual.md @@ -5,12 +5,14 @@ Nim Manual :Authors: Andreas Rumpf, Zahary Karadjov :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: - "Complexity" seems to be a lot like "energy": you can transfer it from the end - user to one/some of the other players, but the total amount seems to remain - pretty much constant for a given task. -- Ran +> "Complexity" seems to be a lot like "energy": you can transfer it from the +> end-user to one/some of the other players, but the total amount seems to remain +> pretty much constant for a given task. -- Ran About this document @@ -20,41 +22,41 @@ About this document precise wording. This manual is constantly evolving into a proper specification. **Note**: The experimental features of Nim are -covered `here <manual_experimental.html>`_. +covered [here](manual_experimental.html). -**Note**: Assignments, moves and destruction are specified in -the `destructors <destructors.html>`_ document. +**Note**: Assignments, moves, and destruction are specified in +the [destructors](destructors.html) document. This document describes the lexis, the syntax, and the semantics of the Nim language. To learn how to compile Nim programs and generate documentation see -`Compiler User Guide <nimc.html>`_ and `DocGen Tools Guide <docgen.html>`_. +the [Compiler User Guide](nimc.html) and the [DocGen Tools Guide](docgen.html). -The language constructs are explained using an extended BNF, in which ``(a)*`` -means 0 or more ``a``'s, ``a+`` means 1 or more ``a``'s, and ``(a)?`` means an +The language constructs are explained using an extended BNF, in which `(a)*` +means 0 or more `a`'s, `a+` means 1 or more `a`'s, and `(a)?` means an optional *a*. Parentheses may be used to group elements. -``&`` is the lookahead operator; ``&a`` means that an ``a`` is expected but +`&` is the lookahead operator; `&a` means that an `a` is expected but not consumed. It will be consumed in the following rule. -The ``|``, ``/`` symbols are used to mark alternatives and have the lowest -precedence. ``/`` is the ordered choice that requires the parser to try the -alternatives in the given order. ``/`` is often used to ensure the grammar +The `|`, `/` symbols are used to mark alternatives and have the lowest +precedence. `/` is the ordered choice that requires the parser to try the +alternatives in the given order. `/` is often used to ensure the grammar is not ambiguous. Non-terminals start with a lowercase letter, abstract terminal symbols are in UPPERCASE. Verbatim terminal symbols (including keywords) are quoted -with ``'``. An example:: +with `'`. An example: - ifStmt = 'if' expr ':' stmts ('elif' expr ':' stmts)* ('else' stmts)? + ifStmt = 'if' expr ':' stmts ('elif' expr ':' stmts)* ('else' stmts)? -The binary ``^*`` operator is used as a shorthand for 0 or more occurrences -separated by its second argument; likewise ``^+`` means 1 or more -occurrences: ``a ^+ b`` is short for ``a (b a)*`` -and ``a ^* b`` is short for ``(a (b a)*)?``. Example:: +The binary `^*` operator is used as a shorthand for 0 or more occurrences +separated by its second argument; likewise `^+` means 1 or more +occurrences: `a ^+ b` is short for `a (b a)*` +and `a ^* b` is short for `(a (b a)*)?`. Example: - arrayConstructor = '[' expr ^* ',' ']' + arrayConstructor = '[' expr ^* ',' ']' Other parts of Nim, like scoping rules or runtime semantics, are described informally. @@ -89,11 +91,11 @@ The nature of this executable depends on the compiler implementation; it may, for example, be a native binary or JavaScript source code. In a typical Nim program, most of the code is compiled into the executable. -However, some of the code may be executed at -`compile time`:idx:. This can include constant expressions, macro definitions, +However, some code may be executed at +`compile-time`:idx:. This can include constant expressions, macro definitions, and Nim procedures used by macro definitions. Most of the Nim language is -supported at compile time, but there are some restrictions -- see `Restrictions -on Compile-Time Execution <#restrictions-on-compileminustime-execution>`_ for +supported at compile-time, but there are some restrictions -- see [Restrictions +on Compile-Time Execution] for details. We use the term `runtime`:idx: to cover both compile-time execution and code execution in the executable. @@ -109,34 +111,35 @@ A `panic`:idx: is an error that the implementation detects and reports at runtime. The method for reporting such errors is via *raising exceptions* or *dying with a fatal error*. However, the implementation provides a means to disable these `runtime checks`:idx:. See the section -pragmas_ for details. +[Pragmas] for details. Whether a panic results in an exception or in a fatal error is -implementation specific. Thus the following program is invalid; even though the +implementation specific. Thus, the following program is invalid; even though the code purports to catch the `IndexDefect` from an out-of-bounds array access, the compiler may instead choose to allow the program to die with a fatal error. -.. code-block:: nim + ```nim var a: array[0..1, char] let i = 5 try: a[i] = 'N' except IndexDefect: echo "invalid index" + ``` -The current implementation allows to switch between these different behaviors -via ``--panics:on|off``. When panics are turned on, the program dies on a +The current implementation allows switching between these different behaviors +via `--panics:on|off`:option:. When panics are turned on, the program dies with a panic, if they are turned off the runtime errors are turned into -exceptions. The benefit of ``--panics:on`` is that it produces smaller binary +exceptions. The benefit of `--panics:on`:option: is that it produces smaller binary code and the compiler has more freedom to optimize the code. An `unchecked runtime error`:idx: is an error that is not guaranteed to be -detected, and can cause the subsequent behavior of the computation to +detected and can cause the subsequent behavior of the computation to be arbitrary. Unchecked runtime errors cannot occur if only `safe`:idx: language features are used and if no runtime checks are disabled. A `constant expression`:idx: is an expression whose value can be computed during -semantic analysis of the code in which it appears. It is never an l-value and +a semantic analysis of the code in which it appears. It is never an l-value and never has side effects. Constant expressions are not limited to the capabilities of semantic analysis, such as constant folding; they can use all Nim language features that are supported for compile-time execution. Since constant @@ -163,7 +166,7 @@ encodings are not supported. Any of the standard platform line termination sequences can be used - the Unix form using ASCII LF (linefeed), the Windows form using the ASCII sequence CR LF (return followed by linefeed), or the old Macintosh form using the ASCII CR (return) character. All of these forms can be -used equally, regardless of platform. +used equally, regardless of the platform. Indentation @@ -180,23 +183,23 @@ lookahead. The parser uses a stack of indentation levels: the stack consists of integers counting the spaces. The indentation information is queried at strategic -places in the parser but ignored otherwise: The pseudo terminal ``IND{>}`` +places in the parser but ignored otherwise: The pseudo-terminal `IND{>}` denotes an indentation that consists of more spaces than the entry at the top -of the stack; ``IND{=}`` an indentation that has the same number of spaces. ``DED`` +of the stack; `IND{=}` an indentation that has the same number of spaces. `DED` is another pseudo terminal that describes the *action* of popping a value -from the stack, ``IND{>}`` then implies to push onto the stack. +from the stack, `IND{>}` then implies to push onto the stack. With this notation we can now easily define the core of the grammar: A block of -statements (simplified example):: +statements (simplified example): - ifStmt = 'if' expr ':' stmt - (IND{=} 'elif' expr ':' stmt)* - (IND{=} 'else' ':' stmt)? + ifStmt = 'if' expr ':' stmt + (IND{=} 'elif' expr ':' stmt)* + (IND{=} 'else' ':' stmt)? - simpleStmt = ifStmt / ... + simpleStmt = ifStmt / ... - stmt = IND{>} stmt ^+ IND{=} DED # list of statements - / simpleStmt # or a simple statement + stmt = IND{>} stmt ^+ IND{=} DED # list of statements + / simpleStmt # or a simple statement @@ -204,23 +207,24 @@ Comments -------- Comments start anywhere outside a string or character literal with the -hash character ``#``. +hash character `#`. Comments consist of a concatenation of `comment pieces`:idx:. A comment piece -starts with ``#`` and runs until the end of the line. The end of line characters +starts with `#` and runs until the end of the line. The end of line characters belong to the piece. If the next line only consists of a comment piece with no other tokens between it and the preceding one, it does not start a new comment: -.. code-block:: nim + ```nim i = 0 # This is a single comment over multiple lines. - # The scanner merges these two pieces. + # The lexer merges these two pieces. # The comment continues here. + ``` -`Documentation comments`:idx: are comments that start with two ``##``. +`Documentation comments`:idx: are comments that start with two `##`. Documentation comments are tokens; they are only allowed at certain places in -the input file as they belong to the syntax tree! +the input file as they belong to the syntax tree. Multiline comments @@ -229,26 +233,41 @@ Multiline comments Starting with version 0.13.0 of the language Nim supports multiline comments. They look like: -.. code-block:: nim + ```nim #[Comment here. Multiple lines are not a problem.]# + ``` Multiline comments support nesting: -.. code-block:: nim + ```nim #[ #[ Multiline comment in already commented out code. ]# proc p[T](x: T) = discard ]# + ``` Multiline documentation comments also exist and support nesting too: -.. code-block:: nim + ```nim proc foo = ##[Long documentation comment - here. + here. ]## + ``` + +You can also use the [discard statement](#statements-and-expressions-discard-statement) together with +[triple quoted string literals](#lexical-analysis-triple-quoted-string-literals) to create multiline comments: + + ```nim + discard """ You can have any Nim code text commented + out inside this with no indentation restrictions. + yes("May I ask a pointless question?") """ + ``` + +This was how multiline comments were done before version 0.13.0, +and it is used to provide specifications to [testament](testament.html#writing-unit-tests) test framework. Identifiers & Keywords @@ -258,22 +277,24 @@ Identifiers in Nim can be any string of letters, digits and underscores, with the following restrictions: * begins with a letter -* does not end with an underscore ``_`` -* two immediate following underscores ``__`` are not allowed:: +* does not end with an underscore `_` +* two immediate following underscores `__` are not allowed: + ``` letter ::= 'A'..'Z' | 'a'..'z' | '\x80'..'\xff' digit ::= '0'..'9' IDENTIFIER ::= letter ( ['_'] (letter | digit) )* + ``` -Currently any Unicode character with an ordinal value > 127 (non ASCII) is -classified as a ``letter`` and may thus be part of an identifier but later +Currently, any Unicode character with an ordinal value > 127 (non-ASCII) is +classified as a `letter` and may thus be part of an identifier but later versions of the language may assign some Unicode characters to belong to the operator characters instead. The following keywords are reserved and cannot be used as identifiers: -.. code-block:: nim - :file: keywords.txt + ```nim file="keywords.txt" + ``` Some keywords are unused; they are reserved for future developments of the language. @@ -284,16 +305,17 @@ Identifier equality Two identifiers are considered equal if the following algorithm returns true: -.. code-block:: nim + ```nim proc sameIdentifier(a, b: string): bool = a[0] == b[0] and a.replace("_", "").toLowerAscii == b.replace("_", "").toLowerAscii + ``` -That means only the first letters are compared in a case sensitive manner. Other -letters are compared case insensitively within the ASCII range and underscores are ignored. +That means only the first letters are compared in a case-sensitive manner. Other +letters are compared case-insensitively within the ASCII range and underscores are ignored. This rather unorthodox way to do identifier comparisons is called -`partial case insensitivity`:idx: and has some advantages over the conventional +`partial case-insensitivity`:idx: and has some advantages over the conventional case sensitivity: It allows programmers to mostly use their own preferred @@ -302,21 +324,49 @@ by different programmers cannot use incompatible conventions. A Nim-aware editor or IDE can show the identifiers as preferred. Another advantage is that it frees the programmer from remembering the exact spelling of an identifier. The exception with respect to the first -letter allows common code like ``var foo: Foo`` to be parsed unambiguously. +letter allows common code like `var foo: Foo` to be parsed unambiguously. -Note that this rule also applies to keywords, meaning that ``notin`` is -the same as ``notIn`` and ``not_in`` (all-lowercase version (``notin``, ``isnot``) +Note that this rule also applies to keywords, meaning that `notin` is +the same as `notIn` and `not_in` (all-lowercase version (`notin`, `isnot`) is the preferred way of writing keywords). Historically, Nim was a fully `style-insensitive`:idx: language. This meant that it was not case-sensitive and underscores were ignored and there was not even a -distinction between ``foo`` and ``Foo``. +distinction between `foo` and `Foo`. + + +Keywords as identifiers +----------------------- + +If a keyword is enclosed in backticks it loses its keyword property and becomes an ordinary identifier. + +Examples + + ```nim + var `var` = "Hello Stropping" + ``` + + ```nim + type Obj = object + `type`: int + + let `object` = Obj(`type`: 9) + assert `object` is Obj + assert `object`.`type` == 9 + + var `var` = 42 + let `let` = 8 + assert `var` + `let` == 50 + + const `assert` = true + assert `assert` + ``` String literals --------------- -Terminal symbol in the grammar: ``STR_LIT``. +Terminal symbol in the grammar: `STR_LIT`. String literals can be delimited by matching double quotes, and can contain the following `escape sequences`:idx:\ : @@ -345,93 +395,93 @@ contain the following `escape sequences`:idx:\ : ``\u`` HHHH `unicode codepoint with hex value HHHH`:idx:; exactly four hex digits are allowed ``\u`` {H+} `unicode codepoint`:idx:; - all hex digits enclosed in ``{}`` are used for + all hex digits enclosed in `{}` are used for the codepoint ================== =================================================== -Strings in Nim may contain any 8-bit value, even embedded zeros. However +Strings in Nim may contain any 8-bit value, even embedded zeros. However, some operations may interpret the first binary zero as a terminator. Triple quoted string literals ----------------------------- -Terminal symbol in the grammar: ``TRIPLESTR_LIT``. +Terminal symbol in the grammar: `TRIPLESTR_LIT`. -String literals can also be delimited by three double quotes -``"""`` ... ``"""``. -Literals in this form may run for several lines, may contain ``"`` and do not +String literals can also be delimited by three double quotes `"""` ... `"""`. +Literals in this form may run for several lines, may contain `"` and do not interpret any escape sequences. -For convenience, when the opening ``"""`` is followed by a newline (there may -be whitespace between the opening ``"""`` and the newline), +For convenience, when the opening `"""` is followed by a newline (there may +be whitespace between the opening `"""` and the newline), the newline (and the preceding whitespace) is not included in the string. The -ending of the string literal is defined by the pattern ``"""[^"]``, so this: +ending of the string literal is defined by the pattern `"""[^"]`, so this: -.. code-block:: nim + ```nim """"long string within quotes"""" + ``` -Produces:: +Produces: - "long string within quotes" + "long string within quotes" Raw string literals ------------------- -Terminal symbol in the grammar: ``RSTR_LIT``. +Terminal symbol in the grammar: `RSTR_LIT`. There are also raw string literals that are preceded with the -letter ``r`` (or ``R``) and are delimited by matching double quotes (just +letter `r` (or `R`) and are delimited by matching double quotes (just like ordinary string literals) and do not interpret the escape sequences. This is especially convenient for regular expressions or Windows paths: -.. code-block:: nim - + ```nim var f = openFile(r"C:\texts\text.txt") # a raw string, so ``\t`` is no tab + ``` -To produce a single ``"`` within a raw string literal, it has to be doubled: - -.. code-block:: nim +To produce a single `"` within a raw string literal, it has to be doubled: + ```nim r"a""b" + ``` -Produces:: +Produces: - a"b + a"b -``r""""`` is not possible with this notation, because the three leading -quotes introduce a triple quoted string literal. ``r"""`` is the same -as ``"""`` since triple quoted string literals do not interpret escape +`r""""` is not possible with this notation, because the three leading +quotes introduce a triple quoted string literal. `r"""` is the same +as `"""` since triple quoted string literals do not interpret escape sequences either. Generalized raw string literals ------------------------------- -Terminal symbols in the grammar: ``GENERALIZED_STR_LIT``, -``GENERALIZED_TRIPLESTR_LIT``. +Terminal symbols in the grammar: `GENERALIZED_STR_LIT`, +`GENERALIZED_TRIPLESTR_LIT`. -The construct ``identifier"string literal"`` (without whitespace between the +The construct `identifier"string literal"` (without whitespace between the identifier and the opening quotation mark) is a generalized raw string literal. It is a shortcut for the construct -``identifier(r"string literal")``, so it denotes a procedure call with a +`identifier(r"string literal")`, so it denotes a routine call with a raw string literal as its only argument. Generalized raw string literals are especially convenient for embedding mini languages directly into Nim (for example regular expressions). -The construct ``identifier"""string literal"""`` exists too. It is a shortcut -for ``identifier("""string literal""")``. +The construct `identifier"""string literal"""` exists too. It is a shortcut +for `identifier("""string literal""")`. Character literals ------------------ -Character literals are enclosed in single quotes ``''`` and can contain the +Character literals are enclosed in single quotes `''` and can contain the same escape sequences as strings - with one exception: the platform dependent `newline`:idx: (``\p``) -is not allowed as it may be wider than one character (often it is the pair -CR/LF for example). Here are the valid `escape sequences`:idx: for character +is not allowed as it may be wider than one character (it can be the pair +CR/LF). Here are the valid `escape sequences`:idx: for character literals: ================== =================================================== @@ -455,107 +505,188 @@ literals: exactly two hex digits are allowed ================== =================================================== -A character is not an Unicode character but a single byte. The reason for this -is efficiency: for the overwhelming majority of use-cases, the resulting -programs will still handle UTF-8 properly as UTF-8 was specially designed for -this. Another reason is that Nim can thus support ``array[char, int]`` or -``set[char]`` efficiently as many algorithms rely on this feature. The `Rune` -type is used for Unicode characters, it can represent any Unicode character. -``Rune`` is declared in the `unicode module <unicode.html>`_. +A character is not a Unicode character but a single byte. +Rationale: It enables the efficient support of `array[char, int]` or +`set[char]`. + +The `Rune` type can represent any Unicode character. +`Rune` is declared in the [unicode module](unicode.html). + +A character literal that does not end in `'` is interpreted as `'` if there +is a preceding backtick token. There must be no whitespace between the preceding +backtick token and the character literal. This special case ensures that a declaration +like ``proc `'customLiteral`(s: string)`` is valid. ``proc `'customLiteral`(s: string)`` +is the same as ``proc `'\''customLiteral`(s: string)``. + +See also [custom numeric literals]. -Numerical constants -------------------- -Numerical constants are of a single type and have the form:: - - hexdigit = digit | 'A'..'F' | 'a'..'f' - octdigit = '0'..'7' - bindigit = '0'..'1' - HEX_LIT = '0' ('x' | 'X' ) hexdigit ( ['_'] hexdigit )* - DEC_LIT = digit ( ['_'] digit )* - OCT_LIT = '0' 'o' octdigit ( ['_'] octdigit )* - BIN_LIT = '0' ('b' | 'B' ) bindigit ( ['_'] bindigit )* - - INT_LIT = HEX_LIT - | DEC_LIT - | OCT_LIT - | BIN_LIT - - INT8_LIT = INT_LIT ['\''] ('i' | 'I') '8' - INT16_LIT = INT_LIT ['\''] ('i' | 'I') '16' - INT32_LIT = INT_LIT ['\''] ('i' | 'I') '32' - INT64_LIT = INT_LIT ['\''] ('i' | 'I') '64' - - UINT_LIT = INT_LIT ['\''] ('u' | 'U') - UINT8_LIT = INT_LIT ['\''] ('u' | 'U') '8' - UINT16_LIT = INT_LIT ['\''] ('u' | 'U') '16' - UINT32_LIT = INT_LIT ['\''] ('u' | 'U') '32' - UINT64_LIT = INT_LIT ['\''] ('u' | 'U') '64' - - exponent = ('e' | 'E' ) ['+' | '-'] digit ( ['_'] digit )* - FLOAT_LIT = digit (['_'] digit)* (('.' digit (['_'] digit)* [exponent]) |exponent) - FLOAT32_SUFFIX = ('f' | 'F') ['32'] - FLOAT32_LIT = HEX_LIT '\'' FLOAT32_SUFFIX - | (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] FLOAT32_SUFFIX - FLOAT64_SUFFIX = ( ('f' | 'F') '64' ) | 'd' | 'D' - FLOAT64_LIT = HEX_LIT '\'' FLOAT64_SUFFIX - | (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] FLOAT64_SUFFIX - - -As can be seen in the productions, numerical constants can contain underscores -for readability. Integer and floating point literals may be given in decimal (no -prefix), binary (prefix ``0b``), octal (prefix ``0o``) and hexadecimal -(prefix ``0x``) notation. - -There exists a literal for each numerical type that is -defined. The suffix starting with an apostrophe ('\'') is called a -`type suffix`:idx:. Literals without a type suffix are of an integer type, -unless the literal contains a dot or ``E|e`` in which case it is of -type ``float``. This integer type is ``int`` if the literal is in the range -``low(i32)..high(i32)``, otherwise it is ``int64``. -For notational convenience the apostrophe of a type suffix -is optional if it is not ambiguous (only hexadecimal floating point literals +Numeric literals +---------------- + +Numeric literals have the form: + + hexdigit = digit | 'A'..'F' | 'a'..'f' + octdigit = '0'..'7' + bindigit = '0'..'1' + unary_minus = '-' # See the section about unary minus + HEX_LIT = unary_minus? '0' ('x' | 'X' ) hexdigit ( ['_'] hexdigit )* + DEC_LIT = unary_minus? digit ( ['_'] digit )* + OCT_LIT = unary_minus? '0' 'o' octdigit ( ['_'] octdigit )* + BIN_LIT = unary_minus? '0' ('b' | 'B' ) bindigit ( ['_'] bindigit )* + + INT_LIT = HEX_LIT + | DEC_LIT + | OCT_LIT + | BIN_LIT + + INT8_LIT = INT_LIT ['\''] ('i' | 'I') '8' + INT16_LIT = INT_LIT ['\''] ('i' | 'I') '16' + INT32_LIT = INT_LIT ['\''] ('i' | 'I') '32' + INT64_LIT = INT_LIT ['\''] ('i' | 'I') '64' + + UINT_LIT = INT_LIT ['\''] ('u' | 'U') + UINT8_LIT = INT_LIT ['\''] ('u' | 'U') '8' + UINT16_LIT = INT_LIT ['\''] ('u' | 'U') '16' + UINT32_LIT = INT_LIT ['\''] ('u' | 'U') '32' + UINT64_LIT = INT_LIT ['\''] ('u' | 'U') '64' + + exponent = ('e' | 'E' ) ['+' | '-'] digit ( ['_'] digit )* + FLOAT_LIT = unary_minus? digit (['_'] digit)* (('.' digit (['_'] digit)* [exponent]) |exponent) + FLOAT32_SUFFIX = ('f' | 'F') ['32'] + FLOAT32_LIT = HEX_LIT '\'' FLOAT32_SUFFIX + | (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] FLOAT32_SUFFIX + FLOAT64_SUFFIX = ( ('f' | 'F') '64' ) | 'd' | 'D' + FLOAT64_LIT = HEX_LIT '\'' FLOAT64_SUFFIX + | (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] FLOAT64_SUFFIX + + CUSTOM_NUMERIC_LIT = (FLOAT_LIT | INT_LIT) '\'' CUSTOM_NUMERIC_SUFFIX + + # CUSTOM_NUMERIC_SUFFIX is any Nim identifier that is not + # a pre-defined type suffix. + + +As can be seen in the productions, numeric literals can contain underscores +for readability. Integer and floating-point literals may be given in decimal (no +prefix), binary (prefix `0b`), octal (prefix `0o`), and hexadecimal +(prefix `0x`) notation. + +The fact that the unary minus `-` in a number literal like `-1` is considered +to be part of the literal is a late addition to the language. The rationale is that +an expression `-128'i8` should be valid and without this special case, this would +be impossible -- `128` is not a valid `int8` value, only `-128` is. + +For the `unary_minus` rule there are further restrictions that are not covered +in the formal grammar. For `-` to be part of the number literal the immediately +preceding character has to be in the +set `{' ', '\t', '\n', '\r', ',', ';', '(', '[', '{'}`. This set was designed to +cover most cases in a natural manner. + +In the following examples, `-1` is a single token: + + ```nim + echo -1 + echo(-1) + echo [-1] + echo 3,-1 + + "abc";-1 + ``` + +In the following examples, `-1` is parsed as two separate tokens +(as `-`:tok: `1`:tok:): + + ```nim + echo x-1 + echo (int)-1 + echo [a]-1 + "abc"-1 + ``` + + +The suffix starting with an apostrophe ('\'') is called a +`type suffix`:idx:. Literals without a type suffix are of an integer type +unless the literal contains a dot or `E|e` in which case it is of +type `float`. This integer type is `int` if the literal is in the range +`low(int32)..high(int32)`, otherwise it is `int64`. +For notational convenience, the apostrophe of a type suffix +is optional if it is not ambiguous (only hexadecimal floating-point literals with a type suffix can be ambiguous). -The type suffixes are: +The pre-defined type suffixes are: ================= ========================= Type Suffix Resulting type of literal ================= ========================= - ``'i8`` int8 - ``'i16`` int16 - ``'i32`` int32 - ``'i64`` int64 - ``'u`` uint - ``'u8`` uint8 - ``'u16`` uint16 - ``'u32`` uint32 - ``'u64`` uint64 - ``'f`` float32 - ``'d`` float64 - ``'f32`` float32 - ``'f64`` float64 + `'i8` int8 + `'i16` int16 + `'i32` int32 + `'i64` int64 + `'u` uint + `'u8` uint8 + `'u16` uint16 + `'u32` uint32 + `'u64` uint64 + `'f` float32 + `'d` float64 + `'f32` float32 + `'f64` float64 ================= ========================= -Floating point literals may also be in binary, octal or hexadecimal +Floating-point literals may also be in binary, octal or hexadecimal notation: -``0B0_10001110100_0000101001000111101011101111111011000101001101001001'f64`` -is approximately 1.72826e35 according to the IEEE floating point standard. +`0B0_10001110100_0000101001000111101011101111111011000101001101001001'f64` +is approximately 1.72826e35 according to the IEEE floating-point standard. -Literals are bounds checked so that they fit the datatype. Non base-10 -literals are used mainly for flags and bit pattern representations, therefore -bounds checking is done on bit width, not value range. If the literal fits in -the bit width of the datatype, it is accepted. +Literals must match the datatype, for example, `333'i8` is an invalid literal. +Non-base-10 literals are used mainly for flags and bit pattern representations, +therefore the checking is done on bit width and not on value range. Hence: 0b10000000'u8 == 0x80'u8 == 128, but, 0b10000000'i8 == 0x80'i8 == -1 instead of causing an overflow error. + +### Custom numeric literals + +If the suffix is not predefined, then the suffix is assumed to be a call +to a proc, template, macro or other callable identifier that is passed the +string containing the literal. The callable identifier needs to be declared +with a special ``'`` prefix: + + ```nim + import std/strutils + type u4 = distinct uint8 # a 4-bit unsigned integer aka "nibble" + proc `'u4`(n: string): u4 = + # The leading ' is required. + result = (parseInt(n) and 0x0F).u4 + + var x = 5'u4 + ``` + +More formally, a custom numeric literal `123'custom` is transformed +to r"123".`'custom` in the parsing step. There is no AST node kind that +corresponds to this transformation. The transformation naturally handles +the case that additional parameters are passed to the callee: + + ```nim + import std/strutils + type u4 = distinct uint8 # a 4-bit unsigned integer aka "nibble" + proc `'u4`(n: string; moreData: int): u4 = + result = (parseInt(n) and 0x0F).u4 + + var x = 5'u4(123) + ``` + +Custom numeric literals are covered by the grammar rule named `CUSTOM_NUMERIC_LIT`. +A custom numeric literal is a single token. + + Operators --------- Nim allows user defined operators. An operator is any combination of the -following characters:: +following characters: = + - * / < > @ $ ~ & % | @@ -565,37 +696,51 @@ following characters:: defined here.) These keywords are also operators: -``and or not xor shl shr div mod in notin is isnot of as from``. +`and or not xor shl shr div mod in notin is isnot of as from`. -`.`:tok: `=`:tok:, `:`:tok:, `::`:tok: are not available as general operators; they +`.`:tok:, `=`:tok:, `:`:tok:, `::`:tok: are not available as general operators; they are used for other notational purposes. -``*:`` is as a special case treated as the two tokens `*`:tok: and `:`:tok: -(to support ``var v*: T``). +`*:` is as a special case treated as the two tokens `*`:tok: and `:`:tok: +(to support `var v*: T`). + +The `not` keyword is always a unary operator, `a not b` is parsed +as `a(not b)`, not as `(a) not (b)`. + +Unicode Operators +----------------- + +These Unicode operators are also parsed as operators: + + ∙ ∘ × ★ ⊗ ⊘ ⊙ ⊛ ⊠⊡ ∩ ∧ ⊓ # same priority as * (multiplication) + ± ⊕ ⊖ ⊞ ⊟ ∪ ∨ ⊔ # same priority as + (addition) + -The ``not`` keyword is always a unary operator, ``a not b`` is parsed -as ``a(not b)``, not as ``(a) not (b)``. +Unicode operators can be combined with non-Unicode operator +symbols. The usual precedence extensions then apply, for example, `⊠=` is an +assignment like operator just like `*=` is. + +No Unicode normalization step is performed. Other tokens ------------ -The following strings denote other tokens:: +The following strings denote other tokens: ` ( ) { } [ ] , ; [. .] {. .} (. .) [: The `slice`:idx: operator `..`:tok: takes precedence over other tokens that -contain a dot: `{..}`:tok: are the three tokens `{`:tok:, `..`:tok:, `}`:tok: +contain a dot: `{..}` are the three tokens `{`:tok:, `..`:tok:, `}`:tok: and not the two tokens `{.`:tok:, `.}`:tok:. - Syntax ====== This section lists Nim's standard syntax. How the parser handles -the indentation is already described in the `Lexical Analysis`_ section. +the indentation is already described in the [Lexical Analysis] section. Nim allows user-definable operators. Binary operators have 11 different levels of precedence. @@ -605,79 +750,92 @@ Binary operators have 11 different levels of precedence. Associativity ------------- -Binary operators whose first character is ``^`` are right-associative, all +Binary operators whose first character is `^` are right-associative, all other binary operators are left-associative. -.. code-block:: nim + ```nim proc `^/`(x, y: float): float = # a right-associative division operator result = x / y echo 12 ^/ 4 ^/ 8 # 24.0 (4 / 8 = 0.5, then 12 / 0.5 = 24.0) echo 12 / 4 / 8 # 0.375 (12 / 4 = 3.0, then 3 / 8 = 0.375) + ``` Precedence ---------- Unary operators always bind stronger than any binary -operator: ``$a + b`` is ``($a) + b`` and not ``$(a + b)``. +operator: `$a + b` is `($a) + b` and not `$(a + b)`. -If an unary operator's first character is ``@`` it is a `sigil-like`:idx: -operator which binds stronger than a ``primarySuffix``: ``@x.abc`` is parsed -as ``(@x).abc`` whereas ``$x.abc`` is parsed as ``$(x.abc)``. +If a unary operator's first character is `@` it is a `sigil-like`:idx: +operator which binds stronger than a `primarySuffix`: `@x.abc` is parsed +as `(@x).abc` whereas `$x.abc` is parsed as `$(x.abc)`. -For binary operators that are not keywords the precedence is determined by the +For binary operators that are not keywords, the precedence is determined by the following rules: -Operators ending in either ``->``, ``~>`` or ``=>`` are called +Operators ending in either `->`, `~>` or `=>` are called `arrow like`:idx:, and have the lowest precedence of all operators. -If the operator ends with ``=`` and its first character is none of -``<``, ``>``, ``!``, ``=``, ``~``, ``?``, it is an *assignment operator* which -has the second lowest precedence. +If the operator ends with `=` and its first character is none of +`<`, `>`, `!`, `=`, `~`, `?`, it is an *assignment operator* which +has the second-lowest precedence. + +Otherwise, precedence is determined by the first character. -Otherwise precedence is determined by the first character. ================ ======================================================= ================== =============== Precedence level Operators First character Terminal symbol ================ ======================================================= ================== =============== - 10 (highest) ``$ ^`` OP10 - 9 ``* / div mod shl shr %`` ``* % \ /`` OP9 - 8 ``+ -`` ``+ - ~ |`` OP8 - 7 ``&`` ``&`` OP7 - 6 ``..`` ``.`` OP6 - 5 ``== <= < >= > != in notin is isnot not of as from`` ``= < > !`` OP5 - 4 ``and`` OP4 - 3 ``or xor`` OP3 - 2 ``@ : ?`` OP2 - 1 *assignment operator* (like ``+=``, ``*=``) OP1 - 0 (lowest) *arrow like operator* (like ``->``, ``=>``) OP0 + 10 (highest) `$ ^` OP10 + 9 `* / div mod shl shr %` `* % \ /` OP9 + 8 `+ -` `+ - ~ |` OP8 + 7 `&` `&` OP7 + 6 `..` `.` OP6 + 5 `== <= < >= > != in notin is isnot not of as from` `= < > !` OP5 + 4 `and` OP4 + 3 `or xor` OP3 + 2 `@ : ?` OP2 + 1 *assignment operator* (like `+=`, `*=`) OP1 + 0 (lowest) *arrow like operator* (like `->`, `=>`) OP0 ================ ======================================================= ================== =============== -Whether an operator is used a prefix operator is also affected by preceding +Whether an operator is used as a prefix operator is also affected by preceding whitespace (this parsing change was introduced with version 0.13.0): -.. code-block:: nim + ```nim echo $foo # is parsed as echo($foo) + ``` -Spacing also determines whether ``(a, b)`` is parsed as an the argument list +Spacing also determines whether `(a, b)` is parsed as an argument list of a call or whether it is parsed as a tuple constructor: -.. code-block:: nim + ```nim echo(1, 2) # pass 1 and 2 to echo + ``` -.. code-block:: nim + ```nim echo (1, 2) # pass the tuple (1, 2) to echo + ``` + +Dot-like operators +------------------ + +Terminal symbol in the grammar: `DOTLIKEOP`. + +Dot-like operators are operators starting with `.`, but not with `..`, for e.g. `.?`; +they have the same precedence as `.`, so that `a.?b.c` is parsed as `(a.?b).c` instead of `a.?(b.c)`. Grammar ------- -The grammar's start symbol is ``module``. +The grammar's start symbol is `module`. .. include:: grammar.txt :literal: @@ -690,9 +848,7 @@ Order of evaluation Order of evaluation is strictly left-to-right, inside-out as it is typical for most others imperative programming languages: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" var s = "" proc p(arg: int): int = @@ -702,14 +858,13 @@ imperative programming languages: discard p(p(1) + p(2)) doAssert s == "123" + ``` Assignments are not special, the left-hand-side expression is evaluated before the right-hand side: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" var v = 0 proc getI(): int = result = v @@ -727,19 +882,18 @@ right-hand side: someCopy(b[getI()], getI()) doAssert b == [1, 0, 0] + ``` Rationale: Consistency with overloaded assignment or assignment-like operations, -``a = b`` can be read as ``performSomeCopy(a, b)``. +`a = b` can be read as `performSomeCopy(a, b)`. However, the concept of "order of evaluation" is only applicable after the code was normalized: The normalization involves template expansions and argument reorderings that have been passed to named parameters: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" var s = "" proc p(): int = @@ -766,6 +920,7 @@ reorderings that have been passed to named parameters: construct(second = q(), first = p()) doAssert s == "qppqpq" + ``` Rationale: This is far easier to implement than hypothetical alternatives. @@ -798,39 +953,39 @@ block. The ability to access and modify compile-time variables adds flexibility to constant expressions that may be surprising to those coming from other statically typed languages. For example, the following code echoes the beginning -of the Fibonacci series **at compile time**. (This is a demonstration of +of the Fibonacci series **at compile-time**. (This is a demonstration of flexibility in defining constants, not a recommended style for solving this -problem!) +problem.) -.. code-block:: nim - :test: "nim c $1" - import strformat + ```nim test = "nim c $1" + import std/strformat - var fib_n {.compileTime.}: int - var fib_prev {.compileTime.}: int - var fib_prev_prev {.compileTime.}: int + var fibN {.compileTime.}: int + var fibPrev {.compileTime.}: int + var fibPrevPrev {.compileTime.}: int - proc next_fib(): int = - result = if fib_n < 2: - fib_n + proc nextFib(): int = + result = if fibN < 2: + fibN else: - fib_prev_prev + fib_prev - inc(fib_n) - fib_prev_prev = fib_prev - fib_prev = result + fibPrevPrev + fibPrev + inc(fibN) + fibPrevPrev = fibPrev + fibPrev = result - const f0 = next_fib() - const f1 = next_fib() + const f0 = nextFib() + const f1 = nextFib() - const display_fib = block: - const f2 = next_fib() + const displayFib = block: + const f2 = nextFib() var result = fmt"Fibonacci sequence: {f0}, {f1}, {f2}" for i in 3..12: - add(result, fmt", {next_fib()}") + add(result, fmt", {nextFib()}") result static: - echo display_fib + echo displayFib + ``` Restrictions on Compile-Time Execution @@ -841,11 +996,11 @@ language features: * methods * closure iterators -* the ``cast`` operator +* the `cast` operator * reference (pointer) types * FFI -The use of wrappers that use FFI and/or ``cast`` is also disallowed. Note that +The use of wrappers that use FFI and/or `cast` is also disallowed. Note that these wrappers include the ones in the standard libraries. Some or all of these restrictions are likely to be lifted over time. @@ -854,7 +1009,7 @@ Some or all of these restrictions are likely to be lifted over time. Types ===== -All expressions have a type which is known during semantic analysis. Nim +All expressions have a type that is known during semantic analysis. Nim is statically typed. One can declare new types, which is in essence defining an identifier that can be used to denote this custom type. @@ -862,7 +1017,7 @@ These are the major type classes: * ordinal types (consist of integer, bool, character, enumeration (and subranges thereof) types) -* floating point types +* floating-point types * string type * structured types * reference (pointer) type @@ -874,18 +1029,18 @@ Ordinal types ------------- Ordinal types have the following characteristics: -- Ordinal types are countable and ordered. This property allows - the operation of functions as ``inc``, ``ord``, ``dec`` on ordinal types to +- Ordinal types are countable and ordered. This property allows the operation + of functions such as `inc`, `ord`, and `dec` on ordinal types to be defined. -- Ordinal values have a smallest possible value. Trying to count further - down than the smallest value produces a panic or a static error. -- Ordinal values have a largest possible value. Trying to count further - than the largest value produces a panic or a static error. +- Ordinal types have a smallest possible value, accessible with `low(type)`. + Trying to count further down than the smallest value produces a panic or + a static error. +- Ordinal types have a largest possible value, accessible with `high(type)`. + Trying to count further up than the largest value produces a panic or + a static error. -Integers, bool, characters and enumeration types (and subranges of these -types) belong to ordinal types. For reasons of simplicity of implementation -the types ``uint`` and ``uint64`` are not ordinal types. (This will be changed -in later versions of the language.) +Integers, bool, characters, and enumeration types (and subranges of these +types) belong to ordinal types. A distinct type is an ordinal type if its base type is an ordinal type. @@ -894,133 +1049,123 @@ Pre-defined integer types ------------------------- These integer types are pre-defined: -``int`` - the generic signed integer type; its size is platform dependent and has the +`int` +: the generic signed integer type; its size is platform-dependent and has the same size as a pointer. This type should be used in general. An integer literal that has no type suffix is of this type if it is in the range - ``low(int32)..high(int32)`` otherwise the literal's type is ``int64``. + `low(int32)..high(int32)` otherwise the literal's type is `int64`. -intXX - additional signed integer types of XX bits use this naming scheme - (example: int16 is a 16 bit wide integer). - The current implementation supports ``int8``, ``int16``, ``int32``, ``int64``. +`int`\ XX +: additional signed integer types of XX bits use this naming scheme + (example: int16 is a 16-bit wide integer). + The current implementation supports `int8`, `int16`, `int32`, `int64`. Literals of these types have the suffix 'iXX. -``uint`` - the generic `unsigned integer`:idx: type; its size is platform dependent and +`uint` +: the generic `unsigned integer`:idx: type; its size is platform-dependent and has the same size as a pointer. An integer literal with the type - suffix ``'u`` is of this type. + suffix `'u` is of this type. -uintXX - additional unsigned integer types of XX bits use this naming scheme - (example: uint16 is a 16 bit wide unsigned integer). - The current implementation supports ``uint8``, ``uint16``, ``uint32``, - ``uint64``. Literals of these types have the suffix 'uXX. +`uint`\ XX +: additional unsigned integer types of XX bits use this naming scheme + (example: uint16 is a 16-bit wide unsigned integer). + The current implementation supports `uint8`, `uint16`, `uint32`, + `uint64`. Literals of these types have the suffix 'uXX. Unsigned operations all wrap around; they cannot lead to over- or underflow errors. In addition to the usual arithmetic operators for signed and unsigned integers -(``+ - *`` etc.) there are also operators that formally work on *signed* +(`+ - *` etc.) there are also operators that formally work on *signed* integers but treat their arguments as *unsigned*: They are mostly provided for backwards compatibility with older versions of the language that lacked unsigned integer types. These unsigned operations for signed integers use -the ``%`` suffix as convention: +the `%` suffix as convention: ====================== ====================================================== operation meaning ====================== ====================================================== -``a +% b`` unsigned integer addition -``a -% b`` unsigned integer subtraction -``a *% b`` unsigned integer multiplication -``a /% b`` unsigned integer division -``a %% b`` unsigned integer modulo operation -``a <% b`` treat ``a`` and ``b`` as unsigned and compare -``a <=% b`` treat ``a`` and ``b`` as unsigned and compare -``ze(a)`` extends the bits of ``a`` with zeros until it has the - width of the ``int`` type -``toU8(a)`` treats ``a`` as unsigned and converts it to an - unsigned integer of 8 bits (but still the - ``int8`` type) -``toU16(a)`` treats ``a`` as unsigned and converts it to an - unsigned integer of 16 bits (but still the - ``int16`` type) -``toU32(a)`` treats ``a`` as unsigned and converts it to an - unsigned integer of 32 bits (but still the - ``int32`` type) +`a +% b` unsigned integer addition +`a -% b` unsigned integer subtraction +`a *% b` unsigned integer multiplication +`a /% b` unsigned integer division +`a %% b` unsigned integer modulo operation +`a <% b` treat `a` and `b` as unsigned and compare +`a <=% b` treat `a` and `b` as unsigned and compare ====================== ====================================================== `Automatic type conversion`:idx: is performed in expressions where different kinds of integer types are used: the smaller type is converted to the larger. A `narrowing type conversion`:idx: converts a larger to a smaller type (for -example ``int32 -> int16``. A `widening type conversion`:idx: converts a -smaller type to a larger type (for example ``int16 -> int32``). In Nim only +example `int32 -> int16`). A `widening type conversion`:idx: converts a +smaller type to a larger type (for example `int16 -> int32`). In Nim only widening type conversions are *implicit*: -.. code-block:: nim + ```nim var myInt16 = 5i16 var myInt: int - myInt16 + 34 # of type ``int16`` - myInt16 + myInt # of type ``int`` - myInt16 + 2i32 # of type ``int32`` + myInt16 + 34 # of type `int16` + myInt16 + myInt # of type `int` + myInt16 + 2i32 # of type `int32` + ``` -However, ``int`` literals are implicitly convertible to a smaller integer type +However, `int` literals are implicitly convertible to a smaller integer type if the literal's value fits this smaller type and such a conversion is less -expensive than other implicit conversions, so ``myInt16 + 34`` produces -an ``int16`` result. +expensive than other implicit conversions, so `myInt16 + 34` produces +an `int16` result. -For further details, see `Convertible relation -<#type-relations-convertible-relation>`_. +For further details, see [Convertible relation]. Subrange types -------------- -A subrange type is a range of values from an ordinal or floating point type (the base +A subrange type is a range of values from an ordinal or floating-point type (the base type). To define a subrange type, one must specify its limiting values -- the lowest and highest value of the type. For example: -.. code-block:: nim + ```nim type Subrange = range[0..5] PositiveFloat = range[0.0..Inf] + Positive* = range[1..high(int)] # as defined in `system` + ``` -``Subrange`` is a subrange of an integer which can only hold the values 0 -to 5. ``PositiveFloat`` defines a subrange of all positive floating point values. -NaN does not belong to any subrange of floating point types. -Assigning any other value to a variable of type ``Subrange`` is a +`Subrange` is a subrange of an integer which can only hold the values 0 +to 5. `PositiveFloat` defines a subrange of all positive floating-point values. +NaN does not belong to any subrange of floating-point types. +Assigning any other value to a variable of type `Subrange` is a panic (or a static error if it can be determined during semantic analysis). Assignments from the base type to one of its subrange types (and vice versa) are allowed. -A subrange type has the same size as its base type (``int`` in the +A subrange type has the same size as its base type (`int` in the Subrange example). -Pre-defined floating point types +Pre-defined floating-point types -------------------------------- -The following floating point types are pre-defined: +The following floating-point types are pre-defined: -``float`` - the generic floating point type; its size used to be platform dependent, - but now it is always mapped to ``float64``. +`float` +: the generic floating-point type; its size used to be platform-dependent, + but now it is always mapped to `float64`. This type should be used in general. -floatXX - an implementation may define additional floating point types of XX bits using - this naming scheme (example: float64 is a 64 bit wide float). The current - implementation supports ``float32`` and ``float64``. Literals of these types +`float`\ XX +: an implementation may define additional floating-point types of XX bits using + this naming scheme (example: `float64` is a 64-bit wide float). The current + implementation supports `float32` and `float64`. Literals of these types have the suffix 'fXX. -Automatic type conversion in expressions with different kinds -of floating point types is performed: See `Convertible relation`_ for further -details. Arithmetic performed on floating point types follows the IEEE -standard. Integer types are not converted to floating point types automatically -and vice versa. +Automatic type conversion in expressions with different kinds of floating-point +types is performed: See [Convertible relation] for further details. Arithmetic +performed on floating-point types follows the IEEE standard. Integer types are +not converted to floating-point types automatically and vice versa. The IEEE standard defines five types of floating-point exceptions: @@ -1044,48 +1189,51 @@ These exceptions inherit from the `FloatingPointDefect`:idx: base class. Nim provides the pragmas `nanChecks`:idx: and `infChecks`:idx: to control whether the IEEE exceptions are ignored or trap a Nim exception: -.. code-block:: nim + ```nim {.nanChecks: on, infChecks: on.} var a = 1.0 var b = 0.0 echo b / b # raises FloatInvalidOpDefect echo a / b # raises FloatOverflowDefect + ``` -In the current implementation ``FloatDivByZeroDefect`` and ``FloatInexactDefect`` -are never raised. ``FloatOverflowDefect`` is raised instead of -``FloatDivByZeroDefect``. +In the current implementation `FloatDivByZeroDefect` and `FloatInexactDefect` +are never raised. `FloatOverflowDefect` is raised instead of +`FloatDivByZeroDefect`. There is also a `floatChecks`:idx: pragma that is a short-cut for the -combination of ``nanChecks`` and ``infChecks`` pragmas. ``floatChecks`` are +combination of `nanChecks` and `infChecks` pragmas. `floatChecks` are turned off as default. -The only operations that are affected by the ``floatChecks`` pragma are -the ``+``, ``-``, ``*``, ``/`` operators for floating point types. +The only operations that are affected by the `floatChecks` pragma are +the `+`, `-`, `*`, `/` operators for floating-point types. An implementation should always use the maximum precision available to evaluate -floating pointer values during semantic analysis; this means expressions like -``0.09'f32 + 0.01'f32 == 0.09'f64 + 0.01'f64`` that are evaluating during +floating-point values during semantic analysis; this means expressions like +`0.09'f32 + 0.01'f32 == 0.09'f64 + 0.01'f64` that are evaluating during constant folding are true. Boolean type ------------ The boolean type is named `bool`:idx: in Nim and can be one of the two -pre-defined values ``true`` and ``false``. Conditions in ``while``, -``if``, ``elif``, ``when``-statements need to be of type ``bool``. +pre-defined values `true` and `false`. Conditions in `while`, +`if`, `elif`, `when`-statements need to be of type `bool`. -This condition holds:: +This condition holds: + ```nim ord(false) == 0 and ord(true) == 1 + ``` -The operators ``not, and, or, xor, <, <=, >, >=, !=, ==`` are defined -for the bool type. The ``and`` and ``or`` operators perform short-cut +The operators `not, and, or, xor, <, <=, >, >=, !=, ==` are defined +for the bool type. The `and` and `or` operators perform short-cut evaluation. Example: -.. code-block:: nim - + ```nim while p != nil and p.name != "xyz": # p.name is not evaluated if p == nil p = p.next + ``` The size of the bool type is one byte. @@ -1093,15 +1241,11 @@ The size of the bool type is one byte. Character type -------------- -The character type is named ``char`` in Nim. Its size is one byte. -Thus it cannot represent an UTF-8 character, but a part of it. -The reason for this is efficiency: for the overwhelming majority of use-cases, -the resulting programs will still handle UTF-8 properly as UTF-8 was specially -designed for this. -Another reason is that Nim can support ``array[char, int]`` or -``set[char]`` efficiently as many algorithms rely on this feature. The -`Rune` type is used for Unicode characters, it can represent any Unicode -character. ``Rune`` is declared in the `unicode module <unicode.html>`_. +The character type is named `char` in Nim. Its size is one byte. +Thus, it cannot represent a UTF-8 character, but a part of it. + +The `Rune` type is used for Unicode characters, it can represent any Unicode +character. `Rune` is declared in the [unicode module](unicode.html). @@ -1111,15 +1255,16 @@ Enumeration types Enumeration types define a new type whose values consist of the ones specified. The values are ordered. Example: -.. code-block:: nim - + ```nim type Direction = enum north, east, south, west + ``` -Now the following holds:: +Now the following holds: + ```nim ord(north) == 0 ord(east) == 1 ord(south) == 2 @@ -1127,10 +1272,11 @@ Now the following holds:: # Also allowed: ord(Direction.west) == 3 + ``` -Thus, north < east < south < west. The comparison operators can be used -with enumeration types. Instead of ``north`` etc, the enum value can also -be qualified with the enum type that it resides in, ``Direction.north``. +The implied order is: north < east < south < west. The comparison operators can be used +with enumeration types. Instead of `north` etc., the enum value can also +be qualified with the enum type that it resides in, `Direction.north`. For better interfacing to other programming languages, the fields of enum types can be assigned an explicit ordinal value. However, the ordinal values @@ -1139,41 +1285,41 @@ explicitly given is assigned the value of the previous field + 1. An explicit ordered enum can have *holes*: -.. code-block:: nim + ```nim type TokenType = enum a = 2, b = 4, c = 89 # holes are valid + ``` -However, it is then not an ordinal anymore, so it is not possible to use these -enums as an index type for arrays. The procedures ``inc``, ``dec``, ``succ`` -and ``pred`` are not available for them either. +However, it is then not ordinal anymore, so it is impossible to use these +enums as an index type for arrays. The procedures `inc`, `dec`, `succ` +and `pred` are not available for them either. -The compiler supports the built-in stringify operator ``$`` for enumerations. +The compiler supports the built-in stringify operator `$` for enumerations. The stringify's result can be controlled by explicitly giving the string values to use: -.. code-block:: nim - + ```nim type MyEnum = enum valueA = (0, "my value A"), valueB = "value B", valueC = 2, valueD = (3, "abc") + ``` As can be seen from the example, it is possible to both specify a field's ordinal value and its string value by using a tuple. It is also possible to only specify one of them. -An enum can be marked with the ``pure`` pragma so that it's fields are -added to a special module specific hidden scope that is only queried +An enum can be marked with the `pure` pragma so that its fields are +added to a special module-specific hidden scope that is only queried as the last attempt. Only non-ambiguous symbols are added to this scope. But one can always access these via type qualification written -as ``MyEnum.value``: - -.. code-block:: nim +as `MyEnum.value`: + ```nim type MyEnum {.pure.} = enum valueA, valueB, valueC, valueD, amb @@ -1185,35 +1331,86 @@ as ``MyEnum.value``: echo valueA # MyEnum.valueA echo amb # Error: Unclear whether it's MyEnum.amb or OtherEnum.amb echo MyEnum.amb # OK. + ``` + +Enum value names are overloadable, much like routines. If both of the enums +`T` and `U` have a member named `foo`, then the identifier `foo` corresponds +to a choice between `T.foo` and `U.foo`. During overload resolution, +the correct type of `foo` is decided from the context. If the type of `foo` is +ambiguous, a static error will be produced. + + ```nim test = "nim c $1" + + type + E1 = enum + value1, + value2 + E2 = enum + value1, + value2 = 4 + + const + Lookuptable = [ + E1.value1: "1", + # no need to qualify value2, known to be E1.value2 + value2: "2" + ] + + proc p(e: E1) = + # disambiguation in 'case' statements: + case e + of value1: echo "A" + of value2: echo "B" + + p value2 + ``` + +In some cases, ambiguity of enums is resolved depending on the relation +between the current scope and the scope the enums were defined in. + + ```nim + # a.nim + type Foo* = enum abc + + # b.nim + import a + type Bar = enum abc + echo abc is Bar # true + + block: + type Baz = enum abc + echo abc is Baz # true + ``` + +To implement bit fields with enums see [Bit fields]. -To implement bit fields with enums see `Bit fields <#set-type-bit-fields>`_ String type ----------- -All string literals are of the type ``string``. A string in Nim is very +All string literals are of the type `string`. A string in Nim is very similar to a sequence of characters. However, strings in Nim are both zero-terminated and have a length field. One can retrieve the length with the -builtin ``len`` procedure; the length never counts the terminating zero. +builtin `len` procedure; the length never counts the terminating zero. The terminating zero cannot be accessed unless the string is converted -to the ``cstring`` type first. The terminating zero assures that this +to the `cstring` type first. The terminating zero assures that this conversion can be done in O(1) and without any allocations. The assignment operator for strings always copies the string. -The ``&`` operator concatenates strings. +The `&` operator concatenates strings. -Most native Nim types support conversion to strings with the special ``$`` proc. -When calling the ``echo`` proc, for example, the built-in stringify operation +Most native Nim types support conversion to strings with the special `$` proc. +When calling the `echo` proc, for example, the built-in stringify operation for the parameter is called: -.. code-block:: nim - + ```nim echo 3 # calls `$` for `int` + ``` Whenever a user creates a specialized object, implementation of this procedure -provides for ``string`` representation. +provides for `string` representation. -.. code-block:: nim + ```nim type Person = object name: string @@ -1225,96 +1422,119 @@ provides for ``string`` representation. # is natively an integer to convert it to # a string " years old." + ``` -While ``$p.name`` can also be used, the ``$`` operation on a string does -nothing. Note that we cannot rely on automatic conversion from an ``int`` to -a ``string`` like we can for the ``echo`` proc. +While `$p.name` can also be used, the `$` operation on a string does +nothing. Note that we cannot rely on automatic conversion from an `int` to +a `string` like we can for the `echo` proc. Strings are compared by their lexicographical order. All comparison operators are available. Strings can be indexed like arrays (lower bound is 0). Unlike arrays, they can be used in case statements: -.. code-block:: nim - + ```nim case paramStr(i) of "-v": incl(options, optVerbose) of "-h", "-?": incl(options, optHelp) else: write(stdout, "invalid command line option!\n") + ``` Per convention, all strings are UTF-8 strings, but this is not enforced. For example, when reading strings from binary files, they are merely a sequence of -bytes. The index operation ``s[i]`` means the i-th *char* of ``s``, not the -i-th *unichar*. The iterator ``runes`` from the `unicode module -<unicode.html>`_ can be used for iteration over all Unicode characters. +bytes. The index operation `s[i]` means the i-th *char* of `s`, not the +i-th *unichar*. The iterator `runes` from the [unicode module](unicode.html) +can be used for iteration over all Unicode characters. cstring type ------------ -The ``cstring`` type meaning `compatible string` is the native representation -of a string for the compilation backend. For the C backend the ``cstring`` type +The `cstring` type meaning `compatible string` is the native representation +of a string for the compilation backend. For the C backend the `cstring` type represents a pointer to a zero-terminated char array -compatible to the type ``char*`` in Ansi C. Its primary purpose lies in easy -interfacing with C. The index operation ``s[i]`` means the i-th *char* of -``s``; however no bounds checking for ``cstring`` is performed making the +compatible with the type `char*` in ANSI C. Its primary purpose lies in easy +interfacing with C. The index operation `s[i]` means the i-th *char* of +`s`; however no bounds checking for `cstring` is performed making the index operation unsafe. -A Nim ``string`` is implicitly convertible -to ``cstring`` for convenience. If a Nim string is passed to a C-style -variadic proc, it is implicitly converted to ``cstring`` too: +A Nim `string` is implicitly convertible +to `cstring` for convenience. If a Nim string is passed to a C-style +variadic proc, it is implicitly converted to `cstring` too: -.. code-block:: nim + ```nim proc printf(formatstr: cstring) {.importc: "printf", varargs, header: "<stdio.h>".} printf("This works %s", "as expected") + ``` Even though the conversion is implicit, it is not *safe*: The garbage collector -does not consider a ``cstring`` to be a root and may collect the underlying -memory. However in practice this almost never happens as the GC considers -stack roots conservatively. One can use the builtin procs ``GC_ref`` and -``GC_unref`` to keep the string data alive for the rare cases where it does -not work. +does not consider a `cstring` to be a root and may collect the underlying +memory. For this reason, the implicit conversion will be removed in future +releases of the Nim compiler. Certain idioms like conversion of a `const` string +to `cstring` are safe and will remain to be allowed. -A `$` proc is defined for cstrings that returns a string. Thus to get a nim +A `$` proc is defined for cstrings that returns a string. Thus, to get a nim string from a cstring: -.. code-block:: nim + ```nim var str: string = "Hello!" var cstr: cstring = str var newstr: string = $cstr + ``` + +`cstring` literals shouldn't be modified. + + ```nim + var x = cstring"literals" + x[1] = 'A' # This is wrong!!! + ``` + +If the `cstring` originates from a regular memory (not read-only memory), +it can be modified: + + ```nim + var x = "123456" + prepareMutation(x) # call `prepareMutation` before modifying the strings + var s: cstring = cstring(x) + s[0] = 'u' # This is ok + ``` + +`cstring` values may also be used in case statements like strings. Structured types ---------------- A variable of a structured type can hold multiple values at the same time. Structured types can be nested to unlimited levels. Arrays, sequences, -tuples, objects and sets belong to the structured types. +tuples, objects, and sets belong to the structured types. Array and sequence types ------------------------ Arrays are a homogeneous type, meaning that each element in the array has the same type. Arrays always have a fixed length specified as a constant expression (except for open arrays). They can be indexed by any ordinal type. -A parameter ``A`` may be an *open array*, in which case it is indexed by -integers from 0 to ``len(A)-1``. An array expression may be constructed by the -array constructor ``[]``. The element type of this array expression is +A parameter `A` may be an *open array*, in which case it is indexed by +integers from 0 to `len(A)-1`. An array expression may be constructed by the +array constructor `[]`. The element type of this array expression is inferred from the type of the first element. All other elements need to be implicitly convertible to this type. +An array type can be defined using the `array[size, T]` syntax, or using +`array[lo..hi, T]` for arrays that start at an index other than zero. + Sequences are similar to arrays but of dynamic length which may change during runtime (like strings). Sequences are implemented as growable arrays, -allocating pieces of memory as items are added. A sequence ``S`` is always -indexed by integers from 0 to ``len(S)-1`` and its bounds are checked. -Sequences can be constructed by the array constructor ``[]`` in conjunction -with the array to sequence operator ``@``. Another way to allocate space for a -sequence is to call the built-in ``newSeq`` procedure. +allocating pieces of memory as items are added. A sequence `S` is always +indexed by integers from 0 to `len(S)-1` and its bounds are checked. +Sequences can be constructed by the array constructor `[]` in conjunction +with the array to sequence operator `@`. Another way to allocate space for a +sequence is to call the built-in `newSeq` procedure. A sequence may be passed to a parameter that is of type *open array*. Example: -.. code-block:: nim - + ```nim type IntArray = array[0..5, int] # an array that is indexed with 0..5 IntSeq = seq[int] # a sequence of integers @@ -1325,25 +1545,25 @@ Example: y = @[1, 2, 3, 4, 5, 6] # the @ turns the array into a sequence let z = [1.0, 2, 3, 4] # the type of z is array[0..3, float] + ``` The lower bound of an array or sequence may be received by the built-in proc -``low()``, the higher bound by ``high()``. The length may be -received by ``len()``. ``low()`` for a sequence or an open array always returns +`low()`, the higher bound by `high()`. The length may be +received by `len()`. `low()` for a sequence or an open array always returns 0, as this is the first valid index. -One can append elements to a sequence with the ``add()`` proc or the ``&`` +One can append elements to a sequence with the `add()` proc or the `&` operator, and remove (and get) the last element of a sequence with the -``pop()`` proc. +`pop()` proc. -The notation ``x[i]`` can be used to access the i-th element of ``x``. +The notation `x[i]` can be used to access the i-th element of `x`. Arrays are always bounds checked (statically or at runtime). These checks can be disabled via pragmas or invoking the compiler with the -``--boundChecks:off`` command line switch. +`--boundChecks:off`:option: command-line switch. An array constructor can have explicit indexes for readability: -.. code-block:: nim - + ```nim type Values = enum valA, valB, valC @@ -1354,12 +1574,12 @@ An array constructor can have explicit indexes for readability: valB: "B", valC: "C" ] + ``` -If an index is left out, ``succ(lastIndex)`` is used as the index +If an index is left out, `succ(lastIndex)` is used as the index value: -.. code-block:: nim - + ```nim type Values = enum valA, valB, valC, valD, valE @@ -1371,6 +1591,7 @@ value: valC: "C", "D", "e" ] + ``` @@ -1379,30 +1600,31 @@ Open arrays Often fixed size arrays turn out to be too inflexible; procedures should be able to deal with arrays of different sizes. The `openarray`:idx: type -allows this; it can only be used for parameters. Openarrays are always -indexed with an ``int`` starting at position 0. The ``len``, ``low`` -and ``high`` operations are available for open arrays too. Any array with -a compatible base type can be passed to an openarray parameter, the index -type does not matter. In addition to arrays sequences can also be passed +allows this; it can only be used for parameters. Open arrays are always +indexed with an `int` starting at position 0. The `len`, `low` +and `high` operations are available for open arrays too. Any array with +a compatible base type can be passed to an open array parameter, the index +type does not matter. In addition to arrays, sequences can also be passed to an open array parameter. -The openarray type cannot be nested: multidimensional openarrays are not +The `openarray` type cannot be nested: multidimensional open arrays are not supported because this is seldom needed and cannot be done efficiently. -.. code-block:: nim + ```nim proc testOpenArray(x: openArray[int]) = echo repr(x) testOpenArray([1,2,3]) # array[] testOpenArray(@[1,2,3]) # seq[] + ``` Varargs ------- -A ``varargs`` parameter is an openarray parameter that additionally -allows to pass a variable number of arguments to a procedure. The compiler +A `varargs` parameter is an open array parameter that additionally +allows a variable number of arguments to be passed to a procedure. The compiler converts the list of arguments to an array implicitly: -.. code-block:: nim + ```nim proc myWriteln(f: File, a: varargs[string]) = for s in items(a): write(f, s) @@ -1411,12 +1633,13 @@ converts the list of arguments to an array implicitly: myWriteln(stdout, "abc", "def", "xyz") # is transformed to: myWriteln(stdout, ["abc", "def", "xyz"]) + ``` -This transformation is only done if the varargs parameter is the +This transformation is only done if the `varargs` parameter is the last parameter in the procedure header. It is also possible to perform type conversions in this context: -.. code-block:: nim + ```nim proc myWriteln(f: File, a: varargs[string, `$`]) = for s in items(a): write(f, s) @@ -1424,52 +1647,57 @@ type conversions in this context: myWriteln(stdout, 123, "abc", 4.0) # is transformed to: - myWriteln(stdout, [$123, $"def", $4.0]) + myWriteln(stdout, [$123, $"abc", $4.0]) + ``` -In this example ``$`` is applied to any argument that is passed to the -parameter ``a``. (Note that ``$`` applied to strings is a nop.) +In this example `$` is applied to any argument that is passed to the +parameter `a`. (Note that `$` applied to strings is a nop.) -Note that an explicit array constructor passed to a ``varargs`` parameter is +Note that an explicit array constructor passed to a `varargs` parameter is not wrapped in another implicit array construction: -.. code-block:: nim + ```nim proc takeV[T](a: varargs[T]) = discard takeV([123, 2, 1]) # takeV's T is "int", not "array of int" + ``` -``varargs[typed]`` is treated specially: It matches a variable list of arguments +`varargs[typed]` is treated specially: It matches a variable list of arguments of arbitrary type but *always* constructs an implicit array. This is required -so that the builtin ``echo`` proc does what is expected: +so that the builtin `echo` proc does what is expected: -.. code-block:: nim + ```nim proc echo*(x: varargs[typed, `$`]) {...} echo @[1, 2, 3] # prints "@[1, 2, 3]" and not "123" + ``` Unchecked arrays ---------------- -The ``UncheckedArray[T]`` type is a special kind of ``array`` where its bounds +The `UncheckedArray[T]` type is a special kind of `array` where its bounds are not checked. This is often useful to implement customized flexibly sized -arrays. Additionally an unchecked array is translated into a C array of +arrays. Additionally, an unchecked array is translated into a C array of undetermined size: -.. code-block:: nim + ```nim type MySeq = object len, cap: int data: UncheckedArray[int] + ``` Produces roughly this C code: -.. code-block:: C + ```C typedef struct { NI len; NI cap; NI data[]; } MySeq; + ``` The base type of the unchecked array may not contain any GC'ed memory but this is currently not checked. @@ -1486,62 +1714,60 @@ A variable of a tuple or object type is a heterogeneous storage container. A tuple or object defines various named *fields* of a type. A tuple also defines a lexicographic *order* of the fields. Tuples are meant to be -heterogeneous storage types with few abstractions. The ``()`` syntax +heterogeneous storage types with few abstractions. The `()` syntax can be used to construct tuples. The order of the fields in the constructor must match the order of the tuple's definition. Different tuple-types are *equivalent* if they specify the same fields of the same type in the same -order. The *names* of the fields also have to be identical. - -The assignment operator for tuples copies each component. -The default assignment operator for objects copies each component. Overloading -of the assignment operator is described `here -<manual_experimental.html#type-bound-operations>`_. - -.. code-block:: nim +order. The *names* of the fields also have to be the same. + ```nim type Person = tuple[name: string, age: int] # type representing a person: - # a person consists of a name - # and an age - var - person: Person + # it consists of a name and an age. + var person: Person person = (name: "Peter", age: 30) - echo person.name + assert person.name == "Peter" # the same, but less readable: person = ("Peter", 30) - echo person[0] + assert person[0] == "Peter" + assert Person is (string, int) + assert (string, int) is Person + assert Person isnot tuple[other: string, age: int] # `other` is a different identifier + ``` A tuple with one unnamed field can be constructed with the parentheses and a trailing comma: -.. code-block:: nim + ```nim proc echoUnaryTuple(a: (int,)) = echo a[0] echoUnaryTuple (1,) + ``` In fact, a trailing comma is allowed for every tuple construction. -The implementation aligns the fields for best access performance. The alignment +The implementation aligns the fields for the best access performance. The alignment is compatible with the way the C compiler does it. -For consistency with ``object`` declarations, tuples in a ``type`` section -can also be defined with indentation instead of ``[]``: +For consistency with `object` declarations, tuples in a `type` section +can also be defined with indentation instead of `[]`: -.. code-block:: nim + ```nim type Person = tuple # type representing a person name: string # a person consists of a name age: Natural # and an age + ``` -Objects provide many features that tuples do not. Object provide inheritance and -the ability to hide fields from other modules. Objects with inheritance enabled -have information about their type at runtime, so that the ``of`` operator can be -used to determine the object's type. The ``of`` -operator is similar to the ``instanceof`` operator in Java. +Objects provide many features that tuples do not. Objects provide inheritance +and the ability to hide fields from other modules. Objects with inheritance +enabled have information about their type at runtime so that the `of` operator +can be used to determine the object's type. The `of` operator is similar to +the `instanceof` operator in Java. -.. code-block:: nim + ```nim type Person = object of RootObj name*: string # the * means that `name` is accessible from other modules @@ -1555,15 +1781,16 @@ operator is similar to the ``instanceof`` operator in Java. person: Person assert(student of Student) # is true assert(student of Person) # also true + ``` -Object fields that should be visible from outside the defining module, have to -be marked by ``*``. In contrast to tuples, different object types are +Object fields that should be visible from outside the defining module have to +be marked by `*`. In contrast to tuples, different object types are never *equivalent*, they are nominal types whereas tuples are structural. -Objects that have no ancestor are implicitly ``final`` and thus have no hidden -type information. One can use the ``inheritable`` pragma to -introduce new object roots apart from ``system.RootObj``. +Objects that have no ancestor are implicitly `final` and thus have no hidden +type information. One can use the `inheritable` pragma to +introduce new object roots apart from `system.RootObj`. -.. code-block:: nim + ```nim type Person = object # example of a final object name*: string @@ -1571,34 +1798,49 @@ introduce new object roots apart from ``system.RootObj``. Student = ref object of Person # Error: inheritance only works with non-final objects id: int + ``` + +The assignment operator for tuples and objects copies each component. +The methods to override this copying behavior are described [here][type +bound operators]. Object construction ------------------- Objects can also be created with an `object construction expression`:idx: that -has the syntax ``T(fieldA: valueA, fieldB: valueB, ...)`` where ``T`` is -an ``object`` type or a ``ref object`` type: +has the syntax `T(fieldA: valueA, fieldB: valueB, ...)` where `T` is +an `object` type or a `ref object` type: -.. code-block:: nim - var student = Student(name: "Anton", age: 5, id: 3) + ```nim + type + Student = object + name: string + age: int + PStudent = ref Student + var a1 = Student(name: "Anton", age: 5) + var a2 = PStudent(name: "Anton", age: 5) + # this also works directly: + var a3 = (ref Student)(name: "Anton", age: 5) + # not all fields need to be mentioned, and they can be mentioned out of order: + var a4 = Student(age: 5) + ``` Note that, unlike tuples, objects require the field names along with their values. -For a ``ref object`` type ``system.new`` is invoked implicitly. +For a `ref object` type `system.new` is invoked implicitly. Object variants --------------- -Often an object hierarchy is overkill in certain situations where simple variant -types are needed. Object variants are tagged unions discriminated via a +Often an object hierarchy is an overkill in certain situations where simple variant +types are needed. Object variants are tagged unions discriminated via an enumerated type used for runtime type flexibility, mirroring the concepts of *sum types* and *algebraic data types (ADTs)* as found in other languages. An example: -.. code-block:: nim - - # This is an example how an abstract syntax tree could be modelled in Nim + ```nim + # This is an example of how an abstract syntax tree could be modelled in Nim type NodeKind = enum # the different node types nkInt, # a leaf with an integer value @@ -1609,7 +1851,7 @@ An example: nkIf # an if statement Node = ref NodeObj NodeObj = object - case kind: NodeKind # the ``kind`` field is the discriminator + case kind: NodeKind # the `kind` field is the discriminator of nkInt: intVal: int of nkFloat: floatVal: float of nkString: strVal: string @@ -1620,11 +1862,11 @@ An example: # create a new case object: var n = Node(kind: nkIf, condition: nil) - # accessing n.thenPart is valid because the ``nkIf`` branch is active: + # accessing n.thenPart is valid because the `nkIf` branch is active: n.thenPart = Node(kind: nkFloat, floatVal: 2.0) # the following statement raises an `FieldDefect` exception, because - # n.kind's value does not fit and the ``nkString`` branch is not active: + # n.kind's value does not fit and the `nkString` branch is not active: n.strVal = "" # invalid: would change the active object branch: @@ -1634,16 +1876,17 @@ An example: rightOp: Node(kind: nkInt, intVal: 2)) # valid: does not change the active object branch: x.kind = nkSub + ``` -As can been seen from the example, an advantage to an object hierarchy is that +As can be seen from the example, an advantage to an object hierarchy is that no casting between different object types is needed. Yet, access to invalid object fields raises an exception. -The syntax of ``case`` in an object declaration follows closely the syntax of -the ``case`` statement: The branches in a ``case`` section may be indented too. +The syntax of `case` in an object declaration follows closely the syntax of +the `case` statement: The branches in a `case` section may be indented too. -In the example the ``kind`` field is called the `discriminator`:idx:\: For -safety its address cannot be taken and assignments to it are restricted: The +In the example, the `kind` field is called the `discriminator`:idx:\: For +safety, its address cannot be taken and assignments to it are restricted: The new value must not lead to a change of the active object branch. Also, when the fields of a particular branch are specified during object construction, the corresponding discriminator value must be specified as a constant expression. @@ -1651,30 +1894,29 @@ corresponding discriminator value must be specified as a constant expression. Instead of changing the active object branch, replace the old object in memory with a new one completely: -.. code-block:: nim - + ```nim var x = Node(kind: nkAdd, leftOp: Node(kind: nkInt, intVal: 4), rightOp: Node(kind: nkInt, intVal: 2)) # change the node's contents: x[] = NodeObj(kind: nkString, strVal: "abc") + ``` -Starting with version 0.20 ``system.reset`` cannot be used anymore to support +Starting with version 0.20 `system.reset` cannot be used anymore to support object branch changes as this never was completely memory safe. -As a special rule, the discriminator kind can also be bounded using a ``case`` +As a special rule, the discriminator kind can also be bounded using a `case` statement. If possible values of the discriminator variable in a -``case`` statement branch are a subset of discriminator values for the selected +`case` statement branch are a subset of discriminator values for the selected object branch, the initialization is considered valid. This analysis only works -for immutable discriminators of an ordinal type and disregards ``elif`` -branches. For discriminator values with a ``range`` type, the compiler +for immutable discriminators of an ordinal type and disregards `elif` +branches. For discriminator values with a `range` type, the compiler checks if the entire range of possible values for the discriminator value is valid for the chosen object branch. A small example: -.. code-block:: nim - + ```nim let unknownKind = nkSub # invalid: unsafe initialization because the kind field is not statically known: @@ -1691,6 +1933,92 @@ A small example: # also valid, since unknownKindBounded can only contain the values nkAdd or nkSub let unknownKindBounded = range[nkAdd..nkSub](unknownKind) z = Node(kind: unknownKindBounded, leftOp: Node(), rightOp: Node()) + ``` + + +cast uncheckedAssign +-------------------- + +Some restrictions for case objects can be disabled via a `{.cast(uncheckedAssign).}` section: + + ```nim test="nim c $1" + type + TokenKind* = enum + strLit, intLit + Token = object + case kind*: TokenKind + of strLit: + s*: string + of intLit: + i*: int64 + + proc passToVar(x: var TokenKind) = discard + + var t = Token(kind: strLit, s: "abc") + + {.cast(uncheckedAssign).}: + # inside the 'cast' section it is allowed to pass 't.kind' to a 'var T' parameter: + passToVar(t.kind) + + # inside the 'cast' section it is allowed to set field 's' even though the + # constructed 'kind' field has an unknown value: + t = Token(kind: t.kind, s: "abc") + + # inside the 'cast' section it is allowed to assign to the 't.kind' field directly: + t.kind = intLit + ``` + +Default values for object fields +-------------------------------- + +Object fields are allowed to have a constant default value. The type of field can be omitted if a default value is given. + +```nim test +type + Foo = object + a: int = 2 + b: float = 3.14 + c = "I can have a default value" + + Bar = ref object + a: int = 2 + b: float = 3.14 + c = "I can have a default value" +``` + +The explicit initialization uses these defaults which includes an `object` created with an object construction expression or the procedure `default`; a `ref object` created with an object construction expression or the procedure `new`; an array or a tuple with a subtype which has a default created with the procedure `default`. + + +```nim test +type + Foo = object + a: int = 2 + b = 3.0 + Bar = ref object + a: int = 2 + b = 3.0 + +block: # created with an object construction expression + let x = Foo() + assert x.a == 2 and x.b == 3.0 + + let y = Bar() + assert y.a == 2 and y.b == 3.0 + +block: # created with an object construction expression + let x = default(Foo) + assert x.a == 2 and x.b == 3.0 + + let y = default(array[1, Foo]) + assert y[0].a == 2 and y[0].b == 3.0 + + let z = default(tuple[x: Foo]) + assert z.x.a == 2 and z.x.b == 3.0 + +block: # created with the procedure `new` + let y = new Bar + assert y.a == 2 and y.b == 3.0 +``` Set type -------- @@ -1705,26 +2033,25 @@ point to and modify the same location in memory (also called `aliasing`:idx:). Nim distinguishes between `traced`:idx: and `untraced`:idx: references. Untraced references are also called *pointers*. Traced references point to -objects of a garbage collected heap, untraced references point to -manually allocated objects or to objects somewhere else in memory. Thus -untraced references are *unsafe*. However for certain low-level operations +objects of a garbage-collected heap, untraced references point to +manually allocated objects or objects somewhere else in memory. Thus, +untraced references are *unsafe*. However, for certain low-level operations (accessing the hardware) untraced references are unavoidable. Traced references are declared with the **ref** keyword, untraced references -are declared with the **ptr** keyword. In general, a `ptr T` is implicitly +are declared with the **ptr** keyword. In general, a `ptr T` is implicitly convertible to the `pointer` type. -An empty subscript ``[]`` notation can be used to derefer a reference, -the ``addr`` procedure returns the address of an item. An address is always +An empty subscript `[]` notation can be used to de-refer a reference, +the `addr` procedure returns the address of an item. An address is always an untraced reference. -Thus the usage of ``addr`` is an *unsafe* feature. +Thus, the usage of `addr` is an *unsafe* feature. -The ``.`` (access a tuple/object field operator) -and ``[]`` (array/string/sequence index operator) operators perform implicit +The `.` (access a tuple/object field operator) +and `[]` (array/string/sequence index operator) operators perform implicit dereferencing operations for reference types: -.. code-block:: nim - + ```nim type Node = ref NodeObj NodeObj = object @@ -1736,79 +2063,77 @@ dereferencing operations for reference types: new(n) n.data = 9 # no need to write n[].data; in fact n[].data is highly discouraged! - -Automatic dereferencing can be performed for the first argument of a routine -call, but this is an experimental feature and is described `here -<manual_experimental.html#type-bound-operations>`_. + ``` In order to simplify structural type checking, recursive tuples are not valid: -.. code-block:: nim + ```nim # invalid recursion type MyTuple = tuple[a: ref MyTuple] + ``` -Likewise ``T = ref T`` is an invalid type. +Likewise `T = ref T` is an invalid type. -As a syntactical extension ``object`` types can be anonymous if -declared in a type section via the ``ref object`` or ``ptr object`` notations. +As a syntactical extension, `object` types can be anonymous if +declared in a type section via the `ref object` or `ptr object` notations. This feature is useful if an object should only gain reference semantics: -.. code-block:: nim - + ```nim type Node = ref object le, ri: Node data: int + ``` -To allocate a new traced object, the built-in procedure ``new`` has to be used. -To deal with untraced memory, the procedures ``alloc``, ``dealloc`` and -``realloc`` can be used. The documentation of the system module contains -further information. +To allocate a new traced object, the built-in procedure `new` has to be used. +To deal with untraced memory, the procedures `alloc`, `dealloc` and +`realloc` can be used. The documentation of the [system](system.html) module +contains further information. Nil --- -If a reference points to *nothing*, it has the value ``nil``. ``nil`` is the -default value for all ``ref`` and ``ptr`` types. The ``nil`` value can also be +If a reference points to *nothing*, it has the value `nil`. `nil` is the +default value for all `ref` and `ptr` types. The `nil` value can also be used like any other literal value. For example, it can be used in an assignment -like ``myRef = nil``. +like `myRef = nil`. -Dereferencing ``nil`` is an unrecoverable fatal runtime error (and not a panic). +Dereferencing `nil` is an unrecoverable fatal runtime error (and not a panic). -A successful dereferencing operation ``p[]`` implies that ``p`` is not nil. This +A successful dereferencing operation `p[]` implies that `p` is not nil. This can be exploited by the implementation to optimize code like: -.. code-block:: nim - + ```nim p[].field = 3 if p != nil: - # if p were nil, ``p[]`` would have caused a crash already, - # so we know ``p`` is always not nil here. + # if p were nil, `p[]` would have caused a crash already, + # so we know `p` is always not nil here. action() + ``` Into: -.. code-block:: nim - + ```nim p[].field = 3 action() + ``` *Note*: This is not comparable to C's "undefined behavior" for dereferencing NULL pointers. -Mixing GC'ed memory with ``ptr`` +Mixing GC'ed memory with `ptr` -------------------------------- Special care has to be taken if an untraced object contains traced objects like -traced references, strings or sequences: in order to free everything properly, -the built-in procedure ``reset`` has to be called before freeing the untraced +traced references, strings, or sequences: in order to free everything properly, +the built-in procedure `reset` has to be called before freeing the untraced memory manually: -.. code-block:: nim + ```nim type Data = tuple[x, y: int, s: string] @@ -1823,44 +2148,43 @@ memory manually: # free the memory: dealloc(d) - -Without the ``reset`` call the memory allocated for the ``d.s`` string would -never be freed. The example also demonstrates two important features for low -level programming: the ``sizeof`` proc returns the size of a type or value -in bytes. The ``cast`` operator can circumvent the type system: the compiler -is forced to treat the result of the ``alloc0`` call (which returns an untyped -pointer) as if it would have the type ``ptr Data``. Casting should only be + ``` + +Without the `reset` call the memory allocated for the `d.s` string would +never be freed. The example also demonstrates two important features for +low-level programming: the `sizeof` proc returns the size of a type or value +in bytes. The `cast` operator can circumvent the type system: the compiler +is forced to treat the result of the `alloc0` call (which returns an untyped +pointer) as if it would have the type `ptr Data`. Casting should only be done if it is unavoidable: it breaks type safety and bugs can lead to mysterious crashes. **Note**: The example only works because the memory is initialized to zero -(``alloc0`` instead of ``alloc`` does this): ``d.s`` is thus initialized to -binary zero which the string assignment can handle. One needs to know low level -details like this when mixing garbage collected data with unmanaged memory. +(`alloc0` instead of `alloc` does this): `d.s` is thus initialized to +binary zero which the string assignment can handle. One needs to know low-level +details like this when mixing garbage-collected data with unmanaged memory. .. XXX finalizers for traced objects Procedural type --------------- -A procedural type is internally a pointer to a procedure. ``nil`` is -an allowed value for variables of a procedural type. Nim uses procedural -types to achieve `functional`:idx: programming techniques. +A procedural type is internally a pointer to a procedure. `nil` is +an allowed value for a variable of a procedural type. Examples: -.. code-block:: nim - + ```nim proc printItem(x: int) = ... proc forEach(c: proc (x: int) {.cdecl.}) = ... forEach(printItem) # this will NOT compile because calling conventions differ + ``` -.. code-block:: nim - + ```nim type OnMouseMove = proc (x, y: int) {.closure.} @@ -1873,90 +2197,90 @@ Examples: # ok, 'onMouseMove' has the default calling convention, which is compatible # to 'closure': setOnMouseMove(onMouseMove) + ``` A subtle issue with procedural types is that the calling convention of the procedure influences the type compatibility: procedural types are only compatible if they have the same calling convention. As a special extension, -a procedure of the calling convention ``nimcall`` can be passed to a parameter -that expects a proc of the calling convention ``closure``. +a procedure of the calling convention `nimcall` can be passed to a parameter +that expects a proc of the calling convention `closure`. Nim supports these `calling conventions`:idx:\: `nimcall`:idx: - is the default convention used for a Nim **proc**. It is the - same as ``fastcall``, but only for C compilers that support ``fastcall``. +: is the default convention used for a Nim **proc**. It is the + same as `fastcall`, but only for C compilers that support `fastcall`. `closure`:idx: - is the default calling convention for a **procedural type** that lacks +: is the default calling convention for a **procedural type** that lacks any pragma annotations. It indicates that the procedure has a hidden implicit parameter (an *environment*). Proc vars that have the calling - convention ``closure`` take up two machine words: One for the proc pointer + convention `closure` take up two machine words: One for the proc pointer and another one for the pointer to implicitly passed environment. `stdcall`:idx: - This is the stdcall convention as specified by Microsoft. The generated C - procedure is declared with the ``__stdcall`` keyword. +: This is the stdcall convention as specified by Microsoft. The generated C + procedure is declared with the `__stdcall` keyword. `cdecl`:idx: - The cdecl convention means that a procedure shall use the same convention +: The cdecl convention means that a procedure shall use the same convention as the C compiler. Under Windows the generated C procedure is declared with - the ``__cdecl`` keyword. + the `__cdecl` keyword. `safecall`:idx: - This is the safecall convention as specified by Microsoft. The generated C - procedure is declared with the ``__safecall`` keyword. The word *safe* +: This is the safecall convention as specified by Microsoft. The generated C + procedure is declared with the `__safecall` keyword. The word *safe* refers to the fact that all hardware registers shall be pushed to the hardware stack. `inline`:idx: - The inline convention means the the caller should not call the procedure, +: The inline convention means the caller should not call the procedure, but inline its code directly. Note that Nim does not inline, but leaves - this to the C compiler; it generates ``__inline`` procedures. This is - only a hint for the compiler: it may completely ignore it and - it may inline procedures that are not marked as ``inline``. + this to the C compiler; it generates `__inline` procedures. This is + only a hint for the compiler: it may completely ignore it, and + it may inline procedures that are not marked as `inline`. `fastcall`:idx: - Fastcall means different things to different C compilers. One gets whatever - the C ``__fastcall`` means. +: Fastcall means different things to different C compilers. One gets whatever + the C `__fastcall` means. `thiscall`:idx: - This is thiscall calling convention as specified by Microsoft, used on C++ - class member functions on the x86 architecture +: This is the thiscall calling convention as specified by Microsoft, used on + C++ class member functions on the x86 architecture. `syscall`:idx: - The syscall convention is the same as ``__syscall`` in C. It is used for +: The syscall convention is the same as `__syscall`:c: in C. It is used for interrupts. `noconv`:idx: - The generated C code will not have any explicit calling convention and thus +: The generated C code will not have any explicit calling convention and thus use the C compiler's default calling convention. This is needed because - Nim's default calling convention for procedures is ``fastcall`` to + Nim's default calling convention for procedures is `fastcall` to improve speed. Most calling conventions exist only for the Windows 32-bit platform. -The default calling convention is ``nimcall``, unless it is an inner proc (a +The default calling convention is `nimcall`, unless it is an inner proc (a proc inside of a proc). For an inner proc an analysis is performed whether it accesses its environment. If it does so, it has the calling convention -``closure``, otherwise it has the calling convention ``nimcall``. +`closure`, otherwise it has the calling convention `nimcall`. Distinct type ------------- -A ``distinct`` type is new type derived from a `base type`:idx: that is +A `distinct` type is a new type derived from a `base type`:idx: that is incompatible with its base type. In particular, it is an essential property of a distinct type that it **does not** imply a subtype relation between it and its base type. Explicit type conversions from a distinct type to its -base type and vice versa are allowed. See also ``distinctBase`` to get the +base type and vice versa are allowed. See also `distinctBase` to get the reverse operation. A distinct type is an ordinal type if its base type is an ordinal type. -Modelling currencies -~~~~~~~~~~~~~~~~~~~~ +### Modeling currencies A distinct type can be used to model different physical `units`:idx: with a numerical base type, for example. The following example models currencies. @@ -1964,7 +2288,7 @@ numerical base type, for example. The following example models currencies. Different currencies should not be mixed in monetary calculations. Distinct types are a perfect tool to model different currencies: -.. code-block:: nim + ```nim type Dollar = distinct int Euro = distinct int @@ -1974,20 +2298,22 @@ types are a perfect tool to model different currencies: e: Euro echo d + 12 - # Error: cannot add a number with no unit and a ``Dollar`` + # Error: cannot add a number with no unit and a `Dollar` + ``` -Unfortunately, ``d + 12.Dollar`` is not allowed either, -because ``+`` is defined for ``int`` (among others), not for ``Dollar``. So -a ``+`` for dollars needs to be defined: +Unfortunately, `d + 12.Dollar` is not allowed either, +because `+` is defined for `int` (among others), not for `Dollar`. So +a `+` for dollars needs to be defined: -.. code-block:: + ```nim proc `+` (x, y: Dollar): Dollar = result = Dollar(int(x) + int(y)) + ``` It does not make sense to multiply a dollar with a dollar, but with a number without unit; and the same holds for division: -.. code-block:: + ```nim proc `*` (x: Dollar, y: int): Dollar = result = Dollar(int(x) * y) @@ -1995,28 +2321,28 @@ number without unit; and the same holds for division: result = Dollar(x * int(y)) proc `div` ... + ``` This quickly gets tedious. The implementations are trivial and the compiler should not generate all this code only to optimize it away later - after all -``+`` for dollars should produce the same binary code as ``+`` for ints. -The pragma `borrow`:idx: has been designed to solve this problem; in principle +`+` for dollars should produce the same binary code as `+` for ints. +The pragma `borrow`:idx: has been designed to solve this problem; in principle, it generates the above trivial implementations: -.. code-block:: nim + ```nim proc `*` (x: Dollar, y: int): Dollar {.borrow.} proc `*` (x: int, y: Dollar): Dollar {.borrow.} proc `div` (x: Dollar, y: int): Dollar {.borrow.} + ``` -The ``borrow`` pragma makes the compiler use the same implementation as +The `borrow` pragma makes the compiler use the same implementation as the proc that deals with the distinct type's base type, so no code is generated. -But it seems all this boilerplate code needs to be repeated for the ``Euro`` -currency. This can be solved with templates_. - -.. code-block:: nim - :test: "nim c $1" +But it seems all this boilerplate code needs to be repeated for the `Euro` +currency. This can be solved with [templates]. + ```nim test = "nim c $1" template additive(typ: typedesc) = proc `+` *(x, y: typ): typ {.borrow.} proc `-` *(x, y: typ): typ {.borrow.} @@ -2045,12 +2371,13 @@ currency. This can be solved with templates_. defineCurrency(Dollar, int) defineCurrency(Euro, int) + ``` The borrow pragma can also be used to annotate the distinct type to allow certain builtin operations to be lifted: -.. code-block:: nim + ```nim type Foo = object a, b: int @@ -2063,19 +2390,19 @@ certain builtin operations to be lifted: # field access now valid bb.a = 90 bb.s = "abc" + ``` -Currently only the dot accessor can be borrowed in this way. +Currently, only the dot accessor can be borrowed in this way. -Avoiding SQL injection attacks -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Avoiding SQL injection attacks An SQL statement that is passed from Nim to an SQL database might be -modelled as a string. However, using string templates and filling in the +modeled as a string. However, using string templates and filling in the values is vulnerable to the famous `SQL injection attack`:idx:\: -.. code-block:: nim - import strutils + ```nim + import std/strutils proc query(db: DbHandle, statement: string) = ... @@ -2084,12 +2411,13 @@ values is vulnerable to the famous `SQL injection attack`:idx:\: db.query("SELECT FROM users WHERE name = '$1'" % username) # Horrible security hole, but the compiler does not mind! + ``` This can be avoided by distinguishing strings that contain SQL from strings that don't. Distinct types provide a means to introduce a new string type -``SQL`` that is incompatible with ``string``: +`SQL` that is incompatible with `string`: -.. code-block:: nim + ```nim type SQL = distinct string @@ -2100,14 +2428,15 @@ that don't. Distinct types provide a means to introduce a new string type db.query("SELECT FROM users WHERE name = '$1'" % username) # Static error: `query` expects an SQL string! + ``` It is an essential property of abstract types that they **do not** imply a subtype relation between the abstract type and its base type. Explicit type -conversions from ``string`` to ``SQL`` are allowed: +conversions from `string` to `SQL` are allowed: -.. code-block:: nim - import strutils, sequtils + ```nim + import std/[strutils, sequtils] proc properQuote(s: string): SQL = # quotes a string properly for an SQL statement @@ -2115,43 +2444,47 @@ conversions from ``string`` to ``SQL`` are allowed: proc `%` (frmt: SQL, values: openarray[string]): SQL = # quote each argument: - let v = values.mapIt(SQL, properQuote(it)) + let v = values.mapIt(properQuote(it)) # we need a temporary type for the type conversion :-( type StrSeq = seq[string] # call strutils.`%`: result = SQL(string(frmt) % StrSeq(v)) db.query("SELECT FROM users WHERE name = '$1'".SQL % [username]) + ``` -Now we have compile-time checking against SQL injection attacks. Since -``"".SQL`` is transformed to ``SQL("")`` no new syntax is needed for nice -looking ``SQL`` string literals. The hypothetical ``SQL`` type actually -exists in the library as the `SqlQuery type <db_common.html#SqlQuery>`_ of -modules like `db_sqlite <db_sqlite.html>`_. +Now we have compile-time checking against SQL injection attacks. Since +`"".SQL` is transformed to `SQL("")` no new syntax is needed for nice +looking `SQL` string literals. The hypothetical `SQL` type actually +exists in the library as the [SqlQuery type](db_common.html#SqlQuery) of +modules like [db_sqlite](db_sqlite.html). Auto type --------- -The ``auto`` type can only be used for return types and parameters. For return +The `auto` type can only be used for return types and parameters. For return types it causes the compiler to infer the type from the routine body: -.. code-block:: nim + ```nim proc returnsInt(): auto = 1984 + ``` For parameters it currently creates implicitly generic routines: -.. code-block:: nim + ```nim proc foo(a, b: auto) = discard + ``` Is the same as: -.. code-block:: nim + ```nim proc foo[T1, T2](a: T1, b: T2) = discard + ``` -However later versions of the language might change this to mean "infer the -parameters' types from the body". Then the above ``foo`` would be rejected as -the parameters' types can not be inferred from an empty ``discard`` statement. +However, later versions of the language might change this to mean "infer the +parameters' types from the body". Then the above `foo` would be rejected as +the parameters' types can not be inferred from an empty `discard` statement. Type relations @@ -2163,121 +2496,36 @@ describe the type checking done by the compiler. Type equality ------------- -Nim uses structural type equivalence for most types. Only for objects, -enumerations and distinct types name equivalence is used. The following -algorithm, *in pseudo-code*, determines type equality: - -.. code-block:: nim - proc typeEqualsAux(a, b: PType, - s: var HashSet[(PType, PType)]): bool = - if (a,b) in s: return true - incl(s, (a,b)) - if a.kind == b.kind: - case a.kind - of int, intXX, float, floatXX, char, string, cstring, pointer, - bool, nil, void: - # leaf type: kinds identical; nothing more to check - result = true - of ref, ptr, var, set, seq, openarray: - result = typeEqualsAux(a.baseType, b.baseType, s) - of range: - result = typeEqualsAux(a.baseType, b.baseType, s) and - (a.rangeA == b.rangeA) and (a.rangeB == b.rangeB) - of array: - result = typeEqualsAux(a.baseType, b.baseType, s) and - typeEqualsAux(a.indexType, b.indexType, s) - of tuple: - if a.tupleLen == b.tupleLen: - for i in 0..a.tupleLen-1: - if not typeEqualsAux(a[i], b[i], s): return false - result = true - of object, enum, distinct: - result = a == b - of proc: - result = typeEqualsAux(a.parameterTuple, b.parameterTuple, s) and - typeEqualsAux(a.resultType, b.resultType, s) and - a.callingConvention == b.callingConvention - - proc typeEquals(a, b: PType): bool = - var s: HashSet[(PType, PType)] = {} - result = typeEqualsAux(a, b, s) - -Since types are graphs which can have cycles, the above algorithm needs an -auxiliary set ``s`` to detect this case. - - -Type equality modulo type distinction -------------------------------------- -The following algorithm (in pseudo-code) determines whether two types -are equal with no respect to ``distinct`` types. For brevity the cycle check -with an auxiliary set ``s`` is omitted: - -.. code-block:: nim - proc typeEqualsOrDistinct(a, b: PType): bool = - if a.kind == b.kind: - case a.kind - of int, intXX, float, floatXX, char, string, cstring, pointer, - bool, nil, void: - # leaf type: kinds identical; nothing more to check - result = true - of ref, ptr, var, set, seq, openarray: - result = typeEqualsOrDistinct(a.baseType, b.baseType) - of range: - result = typeEqualsOrDistinct(a.baseType, b.baseType) and - (a.rangeA == b.rangeA) and (a.rangeB == b.rangeB) - of array: - result = typeEqualsOrDistinct(a.baseType, b.baseType) and - typeEqualsOrDistinct(a.indexType, b.indexType) - of tuple: - if a.tupleLen == b.tupleLen: - for i in 0..a.tupleLen-1: - if not typeEqualsOrDistinct(a[i], b[i]): return false - result = true - of distinct: - result = typeEqualsOrDistinct(a.baseType, b.baseType) - of object, enum: - result = a == b - of proc: - result = typeEqualsOrDistinct(a.parameterTuple, b.parameterTuple) and - typeEqualsOrDistinct(a.resultType, b.resultType) and - a.callingConvention == b.callingConvention - elif a.kind == distinct: - result = typeEqualsOrDistinct(a.baseType, b) - elif b.kind == distinct: - result = typeEqualsOrDistinct(a, b.baseType) +Nim uses structural type equivalence for most types. Only for objects, +enumerations and distinct types and for generic types name equivalence is used. Subtype relation ---------------- -If object ``a`` inherits from ``b``, ``a`` is a subtype of ``b``. This subtype -relation is extended to the types ``var``, ``ref``, ``ptr``: -.. code-block:: nim - proc isSubtype(a, b: PType): bool = - if a.kind == b.kind: - case a.kind - of object: - var aa = a.baseType - while aa != nil and aa != b: aa = aa.baseType - result = aa == b - of var, ref, ptr: - result = isSubtype(a.baseType, b.baseType) +If object `a` inherits from `b`, `a` is a subtype of `b`. -.. XXX nil is a special value! +This subtype relation is extended to the types `var`, `ref`, `ptr`. +If `A` is a subtype of `B` and `A` and `B` are `object` types then: +- `var A` is a subtype of `var B` +- `ref A` is a subtype of `ref B` +- `ptr A` is a subtype of `ptr B`. +**Note**: One of the above pointer-indirections is required for assignment from +a subtype to its parent type to prevent "object slicing". Convertible relation -------------------- -A type ``a`` is **implicitly** convertible to type ``b`` iff the following -algorithm returns true: -.. code-block:: nim +A type `a` is **implicitly** convertible to type `b` iff the following +algorithm returns true: + ```nim proc isImplicitlyConvertible(a, b: PType): bool = - if isSubtype(a, b) or isCovariant(a, b): + if isSubtype(a, b): return true if isIntLiteral(a): return b in {int8, int16, int32, int64, int, uint, uint8, uint16, @@ -2303,37 +2551,46 @@ algorithm returns true: result = b == pointer of string: result = b == cstring + of proc: + result = typeEquals(a, b) or compatibleParametersAndEffects(a, b) + ``` +We used the predicate `typeEquals(a, b)` for the "type equality" property +and the predicate `isSubtype(a, b)` for the "subtype relation". +`compatibleParametersAndEffects(a, b)` is currently not specified. -Implicit conversions are also performed for Nim's ``range`` type +Implicit conversions are also performed for Nim's `range` type constructor. -Let ``a0``, ``b0`` of type ``T``. +Let `a0`, `b0` of type `T`. -Let ``A = range[a0..b0]`` be the argument's type, ``F`` the formal -parameter's type. Then an implicit conversion from ``A`` to ``F`` -exists if ``a0 >= low(F) and b0 <= high(F)`` and both ``T`` and ``F`` +Let `A = range[a0..b0]` be the argument's type, `F` the formal +parameter's type. Then an implicit conversion from `A` to `F` +exists if `a0 >= low(F) and b0 <= high(F)` and both `T` and `F` are signed integers or if both are unsigned integers. -A type ``a`` is **explicitly** convertible to type ``b`` iff the following +A type `a` is **explicitly** convertible to type `b` iff the following algorithm returns true: -.. code-block:: nim + ```nim proc isIntegralType(t: PType): bool = result = isOrdinal(t) or t.kind in {float, float32, float64} proc isExplicitlyConvertible(a, b: PType): bool = result = false if isImplicitlyConvertible(a, b): return true - if typeEqualsOrDistinct(a, b): return true + if typeEquals(a, b): return true + if a == distinct and typeEquals(a.baseType, b): return true + if b == distinct and typeEquals(b.baseType, a): return true if isIntegralType(a) and isIntegralType(b): return true if isSubtype(a, b) or isSubtype(b, a): return true + ``` The convertible relation can be relaxed by a user-defined type `converter`:idx:. -.. code-block:: nim + ```nim converter toInt(x: char): int = result = ord(x) var @@ -2346,53 +2603,68 @@ The convertible relation can be relaxed by a user-defined type # one can use the explicit form too x = chr.toInt echo x # => 97 + ``` -The type conversion ``T(a)`` is an L-value if ``a`` is an L-value and -``typeEqualsOrDistinct(T, typeof(a))`` holds. +The type conversion `T(a)` is an L-value if `a` is an L-value and +`typeEqualsOrDistinct(T, typeof(a))` holds. Assignment compatibility ------------------------ -An expression ``b`` can be assigned to an expression ``a`` iff ``a`` is an -`l-value` and ``isImplicitlyConvertible(b.typ, a.typ)`` holds. +An expression `b` can be assigned to an expression `a` iff `a` is an +`l-value` and `isImplicitlyConvertible(b.typ, a.typ)` holds. + +Overload resolution +=================== -Overloading resolution -====================== +In a call `p(args)` where `p` may refer to more than one +candidate, it is said to be a symbol choice. Overload resolution will attempt to +find the best candidate, thus transforming the symbol choice into a resolved symbol. +The routine `p` that matches best is selected following a series of trials explained below. +In order: Catagory matching, Hierarchical Order Comparison, and finally, Complexity Analysis. -In a call ``p(args)`` the routine ``p`` that matches best is selected. If -multiple routines match equally well, the ambiguity is reported during -semantic analysis. +If multiple candidates match equally well after all trials have been tested, the ambiguity +is reported during semantic analysis. + +First Trial: Catagory matching +-------------------------------- -Every arg in args needs to match. There are multiple different categories how an -argument can match. Let ``f`` be the formal parameter's type and ``a`` the type -of the argument. +Every arg in `args` needs to match and there are multiple different categories of matches. +Let `f` be the formal parameter's type and `a` the type of the argument. -1. Exact match: ``a`` and ``f`` are of the same type. -2. Literal match: ``a`` is an integer literal of value ``v`` - and ``f`` is a signed or unsigned integer type and ``v`` is in ``f``'s - range. Or: ``a`` is a floating point literal of value ``v`` - and ``f`` is a floating point type and ``v`` is in ``f``'s +1. Exact match: `a` and `f` are of the same type. +2. Literal match: `a` is an integer literal of value `v` + and `f` is a signed or unsigned integer type and `v` is in `f`'s + range. Or: `a` is a floating-point literal of value `v` + and `f` is a floating-point type and `v` is in `f`'s range. -3. Generic match: ``f`` is a generic type and ``a`` matches, for - instance ``a`` is ``int`` and ``f`` is a generic (constrained) parameter - type (like in ``[T]`` or ``[T: int|char]``. -4. Subrange or subtype match: ``a`` is a ``range[T]`` and ``T`` - matches ``f`` exactly. Or: ``a`` is a subtype of ``f``. -5. Integral conversion match: ``a`` is convertible to ``f`` and ``f`` and ``a`` - is some integer or floating point type. -6. Conversion match: ``a`` is convertible to ``f``, possibly via a user - defined ``converter``. - -These matching categories have a priority: An exact match is better than a -literal match and that is better than a generic match etc. In the following -``count(p, m)`` counts the number of matches of the matching category ``m`` -for the routine ``p``. - -A routine ``p`` matches better than a routine ``q`` if the following -algorithm returns true:: +3. Generic match: `f` is a generic type and `a` matches, for + instance `a` is `int` and `f` is a generic (constrained) parameter + type (like in `[T]` or `[T: int|char]`). +4. Subrange or subtype match: `a` is a `range[T]` and `T` + matches `f` exactly. Or: `a` is a subtype of `f`. +5. Integral conversion match: `a` is convertible to `f` and `f` and `a` + is some integer or floating-point type. +6. Conversion match: `a` is convertible to `f`, possibly via a user + defined `converter`. + +Each operand may fall into one of the categories above; the operand's +highest priority category. The list above is in order or priority. +If a candidate has more priority matches than all other candidates, it is selected as the +resolved symbol. + +For example, if a candidate with one exact match is compared to a candidate with multiple +generic matches and zero exact matches, the candidate with an exact match will win. + +Below is a pseudocode interpretation of category matching, `count(p, m)` counts the number +of matches of the matching category `m` for the routine `p`. + +A routine `p` matches better than a routine `q` if the following +algorithm returns true: + ```nim for each matching category m in ["exact match", "literal match", "generic match", "subtype match", "integral match", "conversion match"]: @@ -2402,11 +2674,55 @@ algorithm returns true:: else: return false return "ambiguous" + ``` + +Second Trial: Hierarchical Order Comparison +---------------------------------------------- + +The hierarchical order of a type is analogous to its relative specificity. Consider the type defined: + +```nim +type A[T] = object +``` + +Matching formals for this type include `T`, `object`, `A`, `A[...]` and `A[C]` where `C` is a concrete type, `A[...]` +is a generic typeclass composition and `T` is an unconstrained generic type variable. This list is in order of +specificity with respect to `A` as each subsequent category narrows the set of types that are members of their match set. + +In this trail, the formal parameters of candidates are compared in order (1st parameter, 2nd parameter, etc.) to search for +a candidate that has an unrivaled specificity. If such a formal parameter is found, the candidate it belongs to is chosen +as the resolved symbol. + +Third Trial: Complexity Analysis +---------------------------------- + +A slight clarification: While category matching digests all the formal parameters of a candidate at once (order doesn't matter), +specificity comparison and complexity analysis operate on each formal parameter at a time. The following +is the final trial to disambiguate a symbol choice when a pair of formal parameters have the same hierarchical order. + +The complexity of a type is essentially its number of modifiers and depth of shape. The definition with the *highest* +complexity wins. Consider the following types: +```nim +type + A[T] = object + B[T, H] = object +``` -Some examples: +Note: The below examples are not exhaustive. -.. code-block:: nim +We shall say that: + +1. `A[T]` has a higher complexity than `A` +2. `var A[T]` has a higher complexity than `A[T]` +3. `A[A[T]]` has a higher complexity than `A[T]` +4. `B[T, H]` has a higher complexity than `A[T]` (`A` and `B` are not compatible here, but convoluted versions of this exist) +5. `B[ptr T, H]` has a higher complexity than `B[T, H]` + +Some Examples +--------------- + + ```nim proc takesInt(x: int) = echo "int" proc takesInt[T](x: T) = echo "T" proc takesInt(x: int16) = echo "int16" @@ -2418,14 +2734,14 @@ Some examples: takesInt(y) # "int16" var z: range[0..4] = 0 takesInt(z) # "T" + ``` -If this algorithm returns "ambiguous" further disambiguation is performed: -If the argument ``a`` matches both the parameter type ``f`` of ``p`` -and ``g`` of ``q`` via a subtyping relation, the inheritance depth is taken +If the argument `a` matches both the parameter type `f` of `p` +and `g` of `q` via a subtyping relation, the inheritance depth is taken into account: -.. code-block:: nim + ```nim type A = object of RootObj B = object of A @@ -2447,29 +2763,48 @@ into account: # but this is ambiguous: pp(c, c) + ``` -Likewise for generic matches the most specialized generic type (that still +Likewise, for generic matches, the most specialized generic type (that still matches) is preferred: -.. code-block:: nim + ```nim proc gen[T](x: ref ref T) = echo "ref ref T" proc gen[T](x: ref T) = echo "ref T" proc gen[T](x: T) = echo "T" var ri: ref int gen(ri) # "ref T" + ``` + +Type variables match +---------------------- + +When overload resolution is considering candidates, the type variable's definition +is not overlooked as it is used to define the formal parameter's type via variable substitution. + +For example: +```nim +type A +proc p[T: A](param: T) +proc p[T: object](param: T) +``` + +These signatures are not ambiguous for a concrete type of `A` even though the formal parameters match ("T" == "T"). +Instead `T` is treated as a variable in that (`T` ?= `T`) depending on the bound type of `T` at the time of +overload resolution. -Overloading based on 'var T' / 'out T' +Overloading based on 'var T' -------------------------------------- -If the formal parameter ``f`` is of type ``var T`` (or ``out T``) -in addition to the ordinary -type checking, the argument is checked to be an `l-value`:idx:. -``var T`` (or ``out T``) matches better than just ``T`` then. +If the formal parameter `f` is of type `var T` +in addition to the ordinary type checking, +the argument is checked to be an `l-value`:idx:. +`var T` matches better than just `T` then. -.. code-block:: nim + ```nim proc sayHi(x: int): string = # matches a non-var int result = $x @@ -2484,17 +2819,7 @@ type checking, the argument is checked to be an `l-value`:idx:. sayHello(3) # 3 # 13 - - -An l-value matches ``var T`` and ``out T`` equally well, hence -the following is ambiguous: - -.. code-block:: nim - - proc p(x: out string) = x = "" - proc p(x: var string) = x = "" - var v: string - p(v) # ambiguous + ``` Lazy type resolution for untyped @@ -2503,37 +2828,100 @@ Lazy type resolution for untyped **Note**: An `unresolved`:idx: expression is an expression for which no symbol lookups and no type checking have been performed. -Since templates and macros that are not declared as ``immediate`` participate -in overloading resolution it's essential to have a way to pass unresolved -expressions to a template or macro. This is what the meta-type ``untyped`` +Since templates and macros that are not declared as `immediate` participate +in overloading resolution, it's essential to have a way to pass unresolved +expressions to a template or macro. This is what the meta-type `untyped` accomplishes: -.. code-block:: nim + ```nim template rem(x: untyped) = discard rem unresolvedExpression(undeclaredIdentifier) + ``` -A parameter of type ``untyped`` always matches any argument (as long as there is +A parameter of type `untyped` always matches any argument (as long as there is any argument passed to it). But one has to watch out because other overloads might trigger the argument's resolution: -.. code-block:: nim + ```nim template rem(x: untyped) = discard proc rem[T](x: T) = discard # undeclared identifier: 'unresolvedExpression' rem unresolvedExpression(undeclaredIdentifier) + ``` -``untyped`` and ``varargs[untyped]`` are the only metatype that are lazy in this sense, the other -metatypes ``typed`` and ``typedesc`` are not lazy. +`untyped` and `varargs[untyped]` are the only metatype that are lazy in this sense, the other +metatypes `typed` and `typedesc` are not lazy. Varargs matching ---------------- -See `Varargs <#types-varargs>`_. +See [Varargs]. + + +iterable +-------- + +A called `iterator` yielding type `T` can be passed to a template or macro via +a parameter typed as `untyped` (for unresolved expressions) or the type class +`iterable` or `iterable[T]` (after type checking and overload resolution). + + ```nim + iterator iota(n: int): int = + for i in 0..<n: yield i + + template toSeq2[T](a: iterable[T]): seq[T] = + var ret: seq[T] + assert a.typeof is T + for ai in a: ret.add ai + ret + + assert iota(3).toSeq2 == @[0, 1, 2] + assert toSeq2(5..7) == @[5, 6, 7] + assert not compiles(toSeq2(@[1,2])) # seq[int] is not an iterable + assert toSeq2(items(@[1,2])) == @[1, 2] # but items(@[1,2]) is + ``` + + +Overload disambiguation +======================= + +For routine calls "overload resolution" is performed. There is a weaker form of +overload resolution called *overload disambiguation* that is performed when an +overloaded symbol is used in a context where there is additional type information +available. Let `p` be an overloaded symbol. These contexts are: + +- In a function call `q(..., p, ...)` when the corresponding formal parameter + of `q` is a `proc` type. If `q` itself is overloaded then the cartesian product + of every interpretation of `q` and `p` must be considered. +- In an object constructor `Obj(..., field: p, ...)` when `field` is a `proc` + type. Analogous rules exist for array/set/tuple constructors. +- In a declaration like `x: T = p` when `T` is a `proc` type. + +As usual, ambiguous matches produce a compile-time error. + +Named argument overloading +-------------------------- + +Routines with the same type signature can be called individually if +a parameter has different names between them. + + ```Nim + proc foo(x: int) = + echo "Using x: ", x + proc foo(y: int) = + echo "Using y: ", y + + foo(x = 2) # Using x: 2 + foo(y = 2) # Using y: 2 + ``` + +Not supplying the parameter name in such cases results in an +ambiguity error. Statements and expressions @@ -2546,7 +2934,7 @@ statements. Statements are separated into `simple statements`:idx: and `complex statements`:idx:. Simple statements are statements that cannot contain other statements like -assignments, calls or the ``return`` statement; complex statements can +assignments, calls, or the `return` statement; complex statements can contain other statements. To avoid the `dangling else problem`:idx:, complex statements always have to be indented. The details can be found in the grammar. @@ -2555,11 +2943,11 @@ Statement list expression ------------------------- Statements can also occur in an expression context that looks -like ``(stmt1; stmt2; ...; ex)``. This is called -an statement list expression or ``(;)``. The type -of ``(stmt1; stmt2; ...; ex)`` is the type of ``ex``. All the other statements -must be of type ``void``. (One can use ``discard`` to produce a ``void`` type.) -``(;)`` does not introduce a new scope. +like `(stmt1; stmt2; ...; ex)`. This is called +a statement list expression or `(;)`. The type +of `(stmt1; stmt2; ...; ex)` is the type of `ex`. All the other statements +must be of type `void`. (One can use `discard` to produce a `void` type.) +`(;)` does not introduce a new scope. Discard statement @@ -2567,13 +2955,14 @@ Discard statement Example: -.. code-block:: nim + ```nim proc p(x, y: int): int = result = x + y discard p(3, 4) # discard the return value of `p` + ``` -The ``discard`` statement evaluates its expression for side-effects and +The `discard` statement evaluates its expression for side-effects and throws the expression's resulting value away, and should only be used when ignoring this value is known not to cause problems. @@ -2583,54 +2972,70 @@ a static error. The return value can be ignored implicitly if the called proc/iterator has been declared with the `discardable`:idx: pragma: -.. code-block:: nim + ```nim proc p(x, y: int): int {.discardable.} = result = x + y p(3, 4) # now valid + ``` + +however the discardable pragma does not work on templates as templates substitute the AST in place. For example: + + ```nim + {.push discardable .} + template example(): string = "https://nim-lang.org" + {.pop.} -An empty ``discard`` statement is often used as a null statement: + example() + ``` -.. code-block:: nim +This template will resolve into "https://nim-lang.org" which is a string literal and since {.discardable.} doesn't apply to literals, the compiler will error. + +An empty `discard` statement is often used as a null statement: + + ```nim proc classify(s: string) = case s[0] of SymChars, '_': echo "an identifier" of '0'..'9': echo "a number" else: discard + ``` Void context ------------ -In a list of statements every expression except the last one needs to have the -type ``void``. In addition to this rule an assignment to the builtin ``result`` -symbol also triggers a mandatory ``void`` context for the subsequent expressions: +In a list of statements, every expression except the last one needs to have the +type `void`. In addition to this rule an assignment to the builtin `result` +symbol also triggers a mandatory `void` context for the subsequent expressions: -.. code-block:: nim + ```nim proc invalid*(): string = result = "foo" "invalid" # Error: value of type 'string' has to be discarded + ``` -.. code-block:: nim + ```nim proc valid*(): string = let x = 317 "valid" + ``` Var statement ------------- Var statements declare new local and global variables and -initialize them. A comma separated list of variables can be used to specify +initialize them. A comma-separated list of variables can be used to specify variables of the same type: -.. code-block:: nim - + ```nim var a: int = 0 x, y, z: int + ``` -If an initializer is given the type can be omitted: the variable is then of the +If an initializer is given, the type can be omitted: the variable is then of the same type as the initializing expression. Variables are always initialized with a default value if there is no initializing expression. The default value depends on the type and is always a zero in binary. @@ -2644,38 +3049,40 @@ char '\\0' bool false ref or pointer type nil procedural type nil -sequence ``@[]`` -string ``""`` -tuple[x: A, y: B, ...] (default(A), default(B), ...) +sequence `@[]` +string `""` +`tuple[x: A, y: B, ...]` (zeroDefault(A), zeroDefault(B), ...) (analogous for objects) -array[0..., T] [default(T), ...] -range[T] default(T); this may be out of the valid range -T = enum cast[T]\(0); this may be an invalid value +`array[0..., T]` `[zeroDefault(T), ...]` +`range[T]` default(T); this may be out of the valid range +T = enum `cast[T](0)`; this may be an invalid value ============================ ============================================== The implicit initialization can be avoided for optimization reasons with the `noinit`:idx: pragma: -.. code-block:: nim + ```nim var - a {.noInit.}: array[0..1023, char] + a {.noinit.}: array[0..1023, char] + ``` -If a proc is annotated with the ``noinit`` pragma this refers to its implicit -``result`` variable: +If a proc is annotated with the `noinit` pragma, this refers to its implicit +`result` variable: -.. code-block:: nim + ```nim proc returnUndefinedValue: int {.noinit.} = discard + ``` -The implicit initialization can be also prevented by the `requiresInit`:idx: +The implicit initialization can also be prevented by the `requiresInit`:idx: type pragma. The compiler requires an explicit initialization for the object -and all of its fields. However it does a `control flow analysis`:idx: to prove +and all of its fields. However, it does a `control flow analysis`:idx: to prove the variable has been initialized and does not rely on syntactic properties: -.. code-block:: nim + ```nim type - MyObject = object {.requiresInit.} + MyObject {.requiresInit.} = object proc p() = # the following is valid: @@ -2685,36 +3092,118 @@ the variable has been initialized and does not rely on syntactic properties: else: x = a() # use x + ``` +`requiresInit` pragma can also be applied to `distinct` types. + +Given the following distinct type definitions: + + ```nim + type + Foo = object + x: string + + DistinctFoo {.requiresInit, borrow: `.`.} = distinct Foo + DistinctString {.requiresInit.} = distinct string + ``` + +The following code blocks will fail to compile: + + ```nim + var foo: DistinctFoo + foo.x = "test" + doAssert foo.x == "test" + ``` + + ```nim + var s: DistinctString + s = "test" + doAssert string(s) == "test" + ``` + +But these will compile successfully: + + ```nim + let foo = DistinctFoo(Foo(x: "test")) + doAssert foo.x == "test" + ``` + + ```nim + let s = DistinctString("test") + doAssert string(s) == "test" + ``` Let statement ------------- -A ``let`` statement declares new local and global `single assignment`:idx: -variables and binds a value to them. The syntax is the same as that of the ``var`` -statement, except that the keyword ``var`` is replaced by the keyword ``let``. -Let variables are not l-values and can thus not be passed to ``var`` parameters +A `let` statement declares new local and global `single assignment`:idx: +variables and binds a value to them. The syntax is the same as that of the `var` +statement, except that the keyword `var` is replaced by the keyword `let`. +Let variables are not l-values and can thus not be passed to `var` parameters nor can their address be taken. They cannot be assigned new values. -For let variables the same pragmas are available as for ordinary variables. +For let variables, the same pragmas are available as for ordinary variables. + +As `let` statements are immutable after creation they need to define a value +when they are declared. The only exception to this is if the `{.importc.}` +pragma (or any of the other `importX` pragmas) is applied, in this case the +value is expected to come from native code, typically a C/C++ `const`. -As ``let`` statements are immutable after creation they need to define a value -when they are declared. The only exception to this is if the ``{.importc.}`` -pragma (or any of the other ``importX`` pragmas) is applied, in this case the -value is expected to come from native code, typically a C/C++ ``const``. +Special identifier `_` (underscore) +----------------------------------- +The identifier `_` has a special meaning in declarations. +Any definition with the name `_` will not be added to scope, meaning the +definition is evaluated, but cannot be used. As a result the name `_` can be +indefinitely redefined. + + ```nim + let _ = 123 + echo _ # error + let _ = 456 # compiles + ``` Tuple unpacking --------------- -In a ``var`` or ``let`` statement tuple unpacking can be performed. The special -identifier ``_`` can be used to ignore some parts of the tuple: +In a `var`, `let` or `const` statement tuple unpacking can be performed. +The special identifier `_` can be used to ignore some parts of the tuple: + + ```nim + proc returnsTuple(): (int, int, int) = (4, 2, 3) + + let (x, _, z) = returnsTuple() + ``` + +This is treated as syntax sugar for roughly the following: + + ```nim + let + tmpTuple = returnsTuple() + x = tmpTuple[0] + z = tmpTuple[2] + ``` -.. code-block:: nim - proc returnsTuple(): (int, int, int) = (4, 2, 3) +For `var` or `let` statements, if the value expression is a tuple literal, +each expression is directly expanded into an assignment without the use of +a temporary variable. - let (x, _, z) = returnsTuple() + ```nim + let (x, y, z) = (1, 2, 3) + # becomes + let + x = 1 + y = 2 + z = 3 + ``` +Tuple unpacking can also be nested: + + ```nim + proc returnsNestedTuple(): (int, (int, int), int, int) = (4, (5, 7), 2, 3) + + let (x, (_, y), _, z) = returnsNestedTuple() + ``` Const section @@ -2722,16 +3211,25 @@ Const section A const section declares constants whose values are constant expressions: -.. code-block:: - import strutils + ```nim + import std/[strutils] const roundPi = 3.1415 constEval = contains("abc", 'b') # computed at compile time! + ``` Once declared, a constant's symbol can be used as a constant expression. -See `Constants and Constant Expressions <#constants-and-constant-expressions>`_ -for details. +The value part of a constant declaration opens a new scope for each constant, +so no symbols declared in the constant value are accessible outside of it. + + ```nim + const foo = (var a = 1; a) + const bar = a # error + let baz = a # error + ``` + +See [Constants and Constant Expressions] for details. Static statement/expression --------------------------- @@ -2739,14 +3237,27 @@ Static statement/expression A static statement/expression explicitly requires compile-time execution. Even some code that has side effects is permitted in a static block: -.. code-block:: - + ```nim static: echo "echo at compile time" + ``` + +`static` can also be used like a routine. + + ```nim + proc getNum(a: int): int = a + + # Below calls "echo getNum(123)" at compile time. + static: + echo getNum(123) + + # Below call evaluates the "getNum(123)" at compile time, but its + # result gets used at run time. + echo static(getNum(123)) + ``` There are limitations on what Nim code can be executed at compile time; -see `Restrictions on Compile-Time Execution -<#restrictions-on-compileminustime-execution>`_ for details. +see [Restrictions on Compile-Time Execution] for details. It's a static error if the compiler cannot execute the block at compile time. @@ -2756,8 +3267,7 @@ If statement Example: -.. code-block:: nim - + ```nim var name = readLine(stdin) if name == "Andreas": @@ -2766,42 +3276,45 @@ Example: echo "Don't you have a name?" else: echo "Boring name..." - -The ``if`` statement is a simple way to make a branch in the control flow: -The expression after the keyword ``if`` is evaluated, if it is true -the corresponding statements after the ``:`` are executed. Otherwise -the expression after the ``elif`` is evaluated (if there is an -``elif`` branch), if it is true the corresponding statements after -the ``:`` are executed. This goes on until the last ``elif``. If all -conditions fail, the ``else`` part is executed. If there is no ``else`` + ``` + +The `if` statement is a simple way to make a branch in the control flow: +The expression after the keyword `if` is evaluated, if it is true +the corresponding statements after the `:` are executed. Otherwise, +the expression after the `elif` is evaluated (if there is an +`elif` branch), if it is true the corresponding statements after +the `:` are executed. This goes on until the last `elif`. If all +conditions fail, the `else` part is executed. If there is no `else` part, execution continues with the next statement. -In ``if`` statements new scopes begin immediately after -the ``if``/``elif``/``else`` keywords and ends after the +In `if` statements, new scopes begin immediately after +the `if`/`elif`/`else` keywords and ends after the corresponding *then* block. For visualization purposes the scopes have been enclosed -in ``{| |}`` in the following example: +in `{| |}` in the following example: -.. code-block:: nim + ```nim if {| (let m = input =~ re"(\w+)=\w+"; m.isMatch): echo "key ", m[0], " value ", m[1] |} elif {| (let m = input =~ re""; m.isMatch): echo "new m in this scope" |} else: {| echo "m not declared here" |} + ``` Case statement -------------- Example: -.. code-block:: nim - - case readline(stdin) + ```nim + let line = readline(stdin) + case line of "delete-everything", "restart-computer": echo "permission denied" of "go-for-a-walk": echo "please yourself" - else: echo "unknown command" + elif line.len == 0: echo "empty" # optional, must come after `of` branches + else: echo "unknown command" # ditto # indentation of the branches is also allowed; and so is an optional colon # after the selecting expression: @@ -2810,31 +3323,39 @@ Example: echo "permission denied" of "go-for-a-walk": echo "please yourself" else: echo "unknown command" + ``` -The ``case`` statement is similar to the if statement, but it represents -a multi-branch selection. The expression after the keyword ``case`` is +The `case` statement is similar to the `if` statement, but it represents +a multi-branch selection. The expression after the keyword `case` is evaluated and if its value is in a *slicelist* the corresponding statements -(after the ``of`` keyword) are executed. If the value is not in any -given *slicelist* the ``else`` part is executed. If there is no ``else`` -part and not all possible values that ``expr`` can hold occur in a -``slicelist``, a static error occurs. This holds only for expressions of -ordinal types. "All possible values" of ``expr`` are determined by ``expr``'s -type. To suppress the static error an ``else`` part with an -empty ``discard`` statement should be used. - -For non ordinal types it is not possible to list every possible value and so -these always require an ``else`` part. +(after the `of` keyword) are executed. If the value is not in any +given *slicelist*, trailing `elif` and `else` parts are executed using same +semantics as for `if` statement, and `elif` is handled just like `else: if`. +If there are no `else` or `elif` parts and not +all possible values that `expr` can hold occur in a *slicelist*, a static error occurs. +This holds only for expressions of ordinal types. +"All possible values" of `expr` are determined by `expr`'s type. +To suppress the static error an `else: discard` should be used. + +Only ordinal types, floats, strings and cstrings are allowed as values +in case statements. + +For non-ordinal types, it is not possible to list every possible value and so +these always require an `else` part. +An exception to this rule is for the `string` type, which currently doesn't +require a trailing `else` or `elif` branch; it's unspecified whether this will +keep working in future versions. Because case statements are checked for exhaustiveness during semantic analysis, -the value in every ``of`` branch must be a constant expression. +the value in every `of` branch must be a constant expression. This restriction also allows the compiler to generate more performant code. -As a special semantic extension, an expression in an ``of`` branch of a case +As a special semantic extension, an expression in an `of` branch of a case statement may evaluate to a set or array constructor; the set or array is then expanded into a list of its elements: -.. code-block:: nim + ```nim const SymChars: set[char] = {'a'..'z', 'A'..'Z', '\x80'..'\xFF'} @@ -2850,11 +3371,12 @@ expanded into a list of its elements: of 'a'..'z', 'A'..'Z', '\x80'..'\xFF', '_': echo "an identifier" of '0'..'9': echo "a number" else: echo "other" + ``` -The ``case`` statement doesn't produce an l-value, so the following example +The `case` statement doesn't produce an l-value, so the following example won't work: -.. code-block:: nim + ```nim type Foo = ref object x: seq[string] @@ -2869,16 +3391,18 @@ won't work: var foo = Foo(x: @[]) foo.get_x().add("asd") + ``` -This can be fixed by explicitly using ``return``: +This can be fixed by explicitly using `result` or `return`: -.. code-block:: nim + ```nim proc get_x(x: Foo): var seq[string] = case true of true: - return x.x + result = x.x else: - return x.x + result = x.x + ``` When statement @@ -2886,8 +3410,7 @@ When statement Example: -.. code-block:: nim - + ```nim when sizeof(int) == 2: echo "running on a 16 bit system!" elif sizeof(int) == 4: @@ -2896,31 +3419,32 @@ Example: echo "running on a 64 bit system!" else: echo "cannot happen!" + ``` -The ``when`` statement is almost identical to the ``if`` statement with some +The `when` statement is almost identical to the `if` statement with some exceptions: -* Each condition (``expr``) has to be a constant expression (of type ``bool``). +* Each condition (`expr`) has to be a constant expression (of type `bool`). * The statements do not open a new scope. * The statements that belong to the expression that evaluated to true are translated by the compiler, the other statements are not checked for semantics! However, each condition is checked for semantics. -The ``when`` statement enables conditional compilation techniques. As -a special syntactic extension, the ``when`` construct is also available -within ``object`` definitions. +The `when` statement enables conditional compilation techniques. As +a special syntactic extension, the `when` construct is also available +within `object` definitions. When nimvm statement -------------------- -``nimvm`` is a special symbol, that may be used as expression of ``when nimvm`` -statement to differentiate execution path between compile time and the -executable. +`nimvm` is a special symbol that may be used as the expression of a +`when nimvm` statement to differentiate the execution path between +compile-time and the executable. Example: -.. code-block:: nim + ```nim proc someProcThatMayRunInCompileTime(): bool = when nimvm: # This branch is taken at compile time. @@ -2932,15 +3456,16 @@ Example: let rtValue = someProcThatMayRunInCompileTime() assert(ctValue == true) assert(rtValue == false) + ``` -``when nimvm`` statement must meet the following requirements: +A `when nimvm` statement must meet the following requirements: -* Its expression must always be ``nimvm``. More complex expressions are not +* Its expression must always be `nimvm`. More complex expressions are not allowed. -* It must not contain ``elif`` branches. -* It must contain ``else`` branch. +* It must not contain `elif` branches. +* It must contain an `else` branch. * Code in branches must not affect semantics of the code that follows the - ``when nimvm`` statement. E.g. it must not define symbols that are used in + `when nimvm` statement. E.g. it must not define symbols that are used in the following code. Return statement @@ -2948,26 +3473,29 @@ Return statement Example: -.. code-block:: nim - return 40+2 + ```nim + return 40 + 2 + ``` -The ``return`` statement ends the execution of the current procedure. -It is only allowed in procedures. If there is an ``expr``, this is syntactic +The `return` statement ends the execution of the current procedure. +It is only allowed in procedures. If there is an `expr`, this is syntactic sugar for: -.. code-block:: nim + ```nim result = expr return result + ``` -``return`` without an expression is a short notation for ``return result`` if +`return` without an expression is a short notation for `return result` if the proc has a return type. The `result`:idx: variable is always the return value of the procedure. It is automatically declared by the compiler. As all -variables, ``result`` is initialized to (binary) zero: +variables, `result` is initialized to (binary) zero: -.. code-block:: nim + ```nim proc returnZero(): int = # implicitly returns 0 + ``` Yield statement @@ -2975,14 +3503,15 @@ Yield statement Example: -.. code-block:: nim + ```nim yield (1, 2, 3) + ``` -The ``yield`` statement is used instead of the ``return`` statement in +The `yield` statement is used instead of the `return` statement in iterators. It is only valid in iterators. Execution is returned to the body of the for loop that called the iterator. Yield does not end the iteration -process, but execution is passed back to the iterator if the next iteration -starts. See the section about iterators (`Iterators and the for statement`_) +process, but the execution is passed back to the iterator if the next iteration +starts. See the section about iterators ([Iterators and the for statement]) for further information. @@ -2991,7 +3520,7 @@ Block statement Example: -.. code-block:: nim + ```nim var found = false block myblock: for i in 0..3: @@ -3000,11 +3529,12 @@ Example: found = true break myblock # leave the block, in this case both for-loops echo found + ``` -The block statement is a means to group statements to a (named) ``block``. -Inside the block, the ``break`` statement is allowed to leave the block -immediately. A ``break`` statement can contain a name of a surrounding -block to specify which block is to leave. +The block statement is a means to group statements to a (named) `block`. +Inside the block, the `break` statement is allowed to leave the block +immediately. A `break` statement can contain a name of a surrounding +block to specify which block is to be left. Break statement @@ -3012,11 +3542,12 @@ Break statement Example: -.. code-block:: nim + ```nim break + ``` -The ``break`` statement is used to leave a block immediately. If ``symbol`` -is given, it is the name of the enclosing block that is to leave. If it is +The `break` statement is used to leave a block immediately. If `symbol` +is given, it is the name of the enclosing block that is to be left. If it is absent, the innermost block is left. @@ -3025,51 +3556,54 @@ While statement Example: -.. code-block:: nim + ```nim echo "Please tell me your password:" var pw = readLine(stdin) while pw != "12345": echo "Wrong password! Next try:" pw = readLine(stdin) + ``` -The ``while`` statement is executed until the ``expr`` evaluates to false. -Endless loops are no error. ``while`` statements open an `implicit block`, -so that they can be left with a ``break`` statement. +The `while` statement is executed until the `expr` evaluates to false. +Endless loops are no error. `while` statements open an `implicit block` +so that they can be left with a `break` statement. Continue statement ------------------ -A ``continue`` statement leads to the immediate next iteration of the +A `continue` statement leads to the immediate next iteration of the surrounding loop construct. It is only allowed within a loop. A continue statement is syntactic sugar for a nested block: -.. code-block:: nim + ```nim while expr1: stmt1 continue stmt2 + ``` Is equivalent to: -.. code-block:: nim + ```nim while expr1: block myBlockName: stmt1 break myBlockName stmt2 + ``` Assembler statement ------------------- The direct embedding of assembler code into Nim code is supported -by the unsafe ``asm`` statement. Identifiers in the assembler code that refer to +by the unsafe `asm` statement. Identifiers in the assembler code that refer to Nim identifiers shall be enclosed in a special character which can be -specified in the statement's pragmas. The default special character is ``'`'``: +specified in the statement's pragmas. The default special character is `'\`'`: -.. code-block:: nim + ```nim {.push stackTrace:off.} proc addInt(a, b: int): int = # a in eax, and b in edx @@ -3081,10 +3615,11 @@ specified in the statement's pragmas. The default special character is ``'`'``: theEnd: """ {.pop.} + ``` If the GNU assembler is used, quotes and newlines are inserted automatically: -.. code-block:: nim + ```nim proc addInt(a, b: int): int = asm """ addl %%ecx, %%eax @@ -3094,10 +3629,11 @@ If the GNU assembler is used, quotes and newlines are inserted automatically: :"=a"(`result`) :"a"(`a`), "c"(`b`) """ + ``` Instead of: -.. code-block:: nim + ```nim proc addInt(a, b: int): int = asm """ "addl %%ecx, %%eax\n" @@ -3107,23 +3643,25 @@ Instead of: :"=a"(`result`) :"a"(`a`), "c"(`b`) """ + ``` Using statement --------------- -The using statement provides syntactic convenience in modules where +The `using` statement provides syntactic convenience in modules where the same parameter names and types are used over and over. Instead of: -.. code-block:: nim + ```nim proc foo(c: Context; n: Node) = ... proc bar(c: Context; n: Node, counter: int) = ... proc baz(c: Context; n: Node) = ... + ``` One can tell the compiler about the convention that a parameter of -name ``c`` should default to type ``Context``, ``n`` should default to -``Node`` etc.: +name `c` should default to type `Context`, `n` should default to +`Node` etc.: -.. code-block:: nim + ```nim using c: Context n: Node @@ -3137,41 +3675,43 @@ name ``c`` should default to type ``Context``, ``n`` should default to # 'c' is inferred to be of the type 'Context' # 'n' is inferred to be of the type 'Node' # But 'x' and 'y' are of type 'int'. + ``` -The ``using`` section uses the same indentation based grouping syntax as -a ``var`` or ``let`` section. +The `using` section uses the same indentation based grouping syntax as +a `var` or `let` section. -Note that ``using`` is not applied for ``template`` since untyped template -parameters default to the type ``system.untyped``. +Note that `using` is not applied for `template` since the untyped template +parameters default to the type `system.untyped`. -Mixing parameters that should use the ``using`` declaration with parameters +Mixing parameters that should use the `using` declaration with parameters that are explicitly typed is possible and requires a semicolon between them. If expression ------------- -An `if expression` is almost like an if statement, but it is an expression. -This feature is similar to `ternary operators` in other languages. +An `if` expression is almost like an if statement, but it is an expression. +This feature is similar to *ternary operators* in other languages. Example: -.. code-block:: nim + ```nim var y = if x > 8: 9 else: 10 + ``` -An if expression always results in a value, so the ``else`` part is -required. ``Elif`` parts are also allowed. +An `if` expression always results in a value, so the `else` part is +required. `elif` parts are also allowed. When expression --------------- -Just like an `if expression`, but corresponding to the when statement. +Just like an `if` expression, but corresponding to the `when` statement. Case expression --------------- -The `case expression` is again very similar to the case statement: +The `case` expression is again very similar to the case statement: -.. code-block:: nim + ```nim var favoriteFood = case animal of "dog": "bones" of "cat": "mice" @@ -3179,6 +3719,7 @@ The `case expression` is again very similar to the case statement: else: echo "I'm not sure what to serve, but everybody loves ice cream" "ice cream" + ``` As seen in the above example, the case expression can also introduce side effects. When multiple statements are given for a branch, Nim will use @@ -3187,76 +3728,75 @@ the last expression as the result value. Block expression ---------------- -A `block expression` is almost like a block statement, but it is an expression -that uses last expression under the block as the value. +A `block` expression is almost like a block statement, but it is an expression +that uses the last expression under the block as the value. It is similar to the statement list expression, but the statement list expression -does not open new block scope. +does not open a new block scope. -.. code-block:: nim + ```nim let a = block: var fib = @[0, 1] for i in 0..10: fib.add fib[^1] + fib[^2] fib + ``` Table constructor ----------------- A table constructor is syntactic sugar for an array constructor: -.. code-block:: nim + ```nim {"key1": "value1", "key2", "key3": "value2"} # is the same as: [("key1", "value1"), ("key2", "value2"), ("key3", "value2")] + ``` -The empty table can be written ``{:}`` (in contrast to the empty set -which is ``{}``) which is thus another way to write as the empty array -constructor ``[]``. This slightly unusual way of supporting tables +The empty table can be written `{:}` (in contrast to the empty set +which is `{}`) which is thus another way to write the empty array +constructor `[]`. This slightly unusual way of supporting tables has lots of advantages: * The order of the (key,value)-pairs is preserved, thus it is easy to - support ordered dicts with for example ``{key: val}.newOrderedTable``. -* A table literal can be put into a ``const`` section and the compiler + support ordered dicts with for example `{key: val}.newOrderedTable`. +* A table literal can be put into a `const` section and the compiler can easily put it into the executable's data section just like it can for arrays and the generated data section requires a minimal amount of memory. -* Every table implementation is treated equal syntactically. -* Apart from the minimal syntactic sugar the language core does not need to +* Every table implementation is treated equally syntactically. +* Apart from the minimal syntactic sugar, the language core does not need to know about tables. Type conversions ---------------- -Syntactically a `type conversion` is like a procedure call, but a +Syntactically a *type conversion* is like a procedure call, but a type name replaces the procedure name. A type conversion is always safe in the sense that a failure to convert a type to another results in an exception (if it cannot be determined statically). Ordinary procs are often preferred over type conversions in Nim: For instance, -``$`` is the ``toString`` operator by convention and ``toFloat`` and ``toInt`` -can be used to convert from floating point to integer or vice versa. - -A type conversion can also be used to disambiguate overloaded routines: +`$` is the `toString` operator by convention and `toFloat` and `toInt` +can be used to convert from floating-point to integer or vice versa. -.. code-block:: nim +Type conversion can also be used to disambiguate overloaded routines: + ```nim proc p(x: int) = echo "int" proc p(x: string) = echo "string" let procVar = (proc(x: string))(p) procVar("a") + ``` Since operations on unsigned numbers wrap around and are unchecked so are -type conversion to unsigned integers and between unsigned integers. The +type conversions to unsigned integers and between unsigned integers. The rationale for this is mostly better interoperability with the C Programming language when algorithms are ported from C to Nim. -Exception: Values that are converted to an unsigned type at compile time -are checked so that code like ``byte(-1)`` does not compile. - **Note**: Historically the operations were unchecked and the conversions were sometimes checked but starting with the revision 1.0.4 of this document and the language implementation the @@ -3265,28 +3805,44 @@ conversions too are now *always unchecked*. Type casts ---------- -Example: -.. code-block:: nim +*Type casts* are a crude mechanism to interpret the bit pattern of an expression +as if it would be of another type. Type casts are only needed for low-level +programming and are inherently unsafe. + + ```nim cast[int](x) + ``` + +The target type of a cast must be a concrete type, for instance, a target type +that is a type class (which is non-concrete) would be invalid: + + ```nim + type Foo = int or float + var x = cast[Foo](1) # Error: cannot cast to a non concrete type: 'Foo' + ``` -Type casts are a crude mechanism to interpret the bit pattern of -an expression as if it would be of another type. Type casts are -only needed for low-level programming and are inherently unsafe. +Type casts should not be confused with *type conversions,* as mentioned in the +prior section. Unlike type conversions, a type cast cannot change the underlying +bit pattern of the data being cast (aside from that the size of the target type +may differ from the source type). Casting resembles *type punning* in other +languages or C++'s `reinterpret_cast`:cpp: and `bit_cast`:cpp: features. +If the size of the target type is larger than the size of the source type, +the remaining memory is zeroed. The addr operator ----------------- -The ``addr`` operator returns the address of an l-value. If the type of the -location is ``T``, the `addr` operator result is of the type ``ptr T``. An +The `addr` operator returns the address of an l-value. If the type of the +location is `T`, the `addr` operator result is of the type `ptr T`. An address is always an untraced reference. Taking the address of an object that resides on the stack is **unsafe**, as the pointer may live longer than the object on the stack and can thus reference a non-existing object. One can get -the address of variables, but one can't use it on variables declared through -``let`` statements: - -.. code-block:: nim +the address of variables. For easier interoperability with other compiled languages +such as C, retrieving the address of a `let` variable, a parameter, +or a `for` loop variable can be accomplished too: + ```nim let t1 = "Hello" var t2 = t1 @@ -3295,23 +3851,19 @@ the address of variables, but one can't use it on variables declared through # --> ref 0x7fff6b71b670 --> 0x10bb81050"Hello" echo cast[ptr string](t3)[] # --> Hello - # The following line doesn't compile: + # The following line also works echo repr(addr(t1)) - # Error: expression has no address - + ``` The unsafeAddr operator ----------------------- -For easier interoperability with other compiled languages such as C, retrieving -the address of a ``let`` variable, a parameter or a ``for`` loop variable, the -``unsafeAddr`` operation can be used: - -.. code-block:: nim +The `unsafeAddr` operator is a deprecated alias for the `addr` operator: + ```nim let myArray = [1, 2, 3] foreignProcThatTakesAnAddr(unsafeAddr myArray) - + ``` Procedures ========== @@ -3321,12 +3873,12 @@ called `procedures`:idx: in Nim. A procedure declaration consists of an identifier, zero or more formal parameters, a return value type and a block of code. Formal parameters are declared as a list of identifiers separated by either comma or semicolon. A parameter is given a type -by ``: typename``. The type applies to all parameters immediately before it, -until either the beginning of the parameter list, a semicolon separator or an +by `: typename`. The type applies to all parameters immediately before it, +until either the beginning of the parameter list, a semicolon separator, or an already typed parameter, is reached. The semicolon can be used to make separation of types and subsequent identifiers more distinct. -.. code-block:: nim + ```nim # Using only commas proc foo(a, b: int, c, d: bool): int @@ -3335,31 +3887,34 @@ separation of types and subsequent identifiers more distinct. # Will fail: a is untyped since ';' stops type propagation. proc foo(a; b: int; c, d: bool): int + ``` A parameter may be declared with a default value which is used if the caller -does not provide a value for the argument. +does not provide a value for the argument. The value will be reevaluated +every time the function is called. -.. code-block:: nim - # b is optional with 47 as its default value + ```nim + # b is optional with 47 as its default value. proc foo(a: int, b: int = 47): int + ``` Parameters can be declared mutable and so allow the proc to modify those arguments, by using the type modifier `var`. -.. code-block:: nim + ```nim # "returning" a value to the caller through the 2nd argument # Notice that the function uses no actual return value at all (ie void) proc foo(inp: int, outp: var int) = outp = inp + 47 + ``` -If the proc declaration has no body, it is a `forward`:idx: declaration. If the -proc returns a value, the procedure body can access an implicitly declared +If the proc declaration doesn't have a body, it is a `forward`:idx: declaration. +If the proc returns a value, the procedure body can access an implicitly declared variable named `result`:idx: that represents the return value. Procs can be overloaded. The overloading resolution algorithm determines which proc is the best match for the arguments. Example: -.. code-block:: nim - + ```nim proc toLower(c: char): char = # toLower for characters if c in {'A'..'Z'}: result = chr(ord(c) + (ord('a') - ord('A'))) @@ -3370,10 +3925,11 @@ best match for the arguments. Example: result = newString(len(s)) for i in 0..len(s) - 1: result[i] = toLower(s[i]) # calls toLower for characters; no recursion! + ``` -Calling a procedure can be done in many different ways: +Calling a procedure can be done in many ways: -.. code-block:: nim + ```nim proc callme(x, y: int, s: string = "", c: char, b: bool = false) = ... # call with positional arguments # parameter bindings: @@ -3384,16 +3940,18 @@ Calling a procedure can be done in many different ways: callme(c='\t', y=1, x=0) # (x=0, y=1, s="", c='\t', b=false) # call as a command statement: no () needed: callme 0, 1, "abc", '\t' # (x=0, y=1, s="abc", c='\t', b=false) + ``` A procedure may call itself recursively. `Operators`:idx: are procedures with a special operator symbol as identifier: -.. code-block:: nim + ```nim proc `$` (x: int): string = # converts an integer to a string; this is a prefix operator. result = intToStr(x) + ``` Operators with one parameter are prefix operators, operators with two parameters are infix operators. (However, the parser distinguishes these from @@ -3401,15 +3959,16 @@ the operator's position within an expression.) There is no way to declare postfix operators: all postfix operators are built-in and handled by the grammar explicitly. -Any operator can be called like an ordinary proc with the '`opr`' +Any operator can be called like an ordinary proc with the \`opr\` notation. (Thus an operator can have more than two parameters): -.. code-block:: nim + ```nim proc `*+` (a, b, c: int): int = # Multiply and add result = a * b + c assert `*+`(3, 4, 6) == `+`(`*`(a, b), c) + ``` Export marker @@ -3418,8 +3977,7 @@ Export marker If a declared symbol is marked with an `asterisk`:idx: it is exported from the current module: -.. code-block:: nim - + ```nim proc exportedEcho*(s: string) = echo s proc `*`*(a: string; b: int): string = result = newStringOfCap(a.len * b) @@ -3430,38 +3988,38 @@ current module: type ExportedType* = object exportedField*: int + ``` Method call syntax ------------------ -For object oriented programming, the syntax ``obj.method(args)`` can be used -instead of ``method(obj, args)``. The parentheses can be omitted if there are no -remaining arguments: ``obj.len`` (instead of ``len(obj)``). +For object-oriented programming, the syntax `obj.methodName(args)` can be used +instead of `methodName(obj, args)`. The parentheses can be omitted if +there are no remaining arguments: `obj.len` (instead of `len(obj)`). This method call syntax is not restricted to objects, it can be used to supply any type of first argument for procedures: -.. code-block:: nim - + ```nim echo "abc".len # is the same as echo len "abc" echo "abc".toUpper() echo {'a', 'b', 'c'}.card stdout.writeLine("Hallo") # the same as writeLine(stdout, "Hallo") + ``` Another way to look at the method call syntax is that it provides the missing postfix notation. The method call syntax conflicts with explicit generic instantiations: -``p[T](x)`` cannot be written as ``x.p[T]`` because ``x.p[T]`` is always -parsed as ``(x.p)[T]``. +`p[T](x)` cannot be written as `x.p[T]` because `x.p[T]` is always +parsed as `(x.p)[T]`. -See also: `Limitations of the method call syntax -<#templates-limitations-of-the-method-call-syntax>`_. +See also: [Limitations of the method call syntax]. -The ``[: ]`` notation has been designed to mitigate this issue: ``x.p[:T]`` -is rewritten by the parser to ``p[T](x)``, ``x.p[:T](y)`` is rewritten to -``p[T](x, y)``. Note that ``[: ]`` has no AST representation, the rewrite +The `[: ]` notation has been designed to mitigate this issue: `x.p[:T]` +is rewritten by the parser to `p[T](x)`, `x.p[:T](y)` is rewritten to +`p[T](x, y)`. Note that `[: ]` has no AST representation, the rewrite is performed directly in the parsing step. @@ -3469,9 +4027,9 @@ Properties ---------- Nim has no need for *get-properties*: Ordinary get-procedures that are called with the *method call syntax* achieve the same. But setting a value is -different; for this a special setter syntax is needed: +different; for this, a special setter syntax is needed: -.. code-block:: nim + ```nim # Module asocket type Socket* = ref object of RootObj @@ -3480,55 +4038,57 @@ different; for this a special setter syntax is needed: proc `host=`*(s: var Socket, value: int) {.inline.} = ## setter of hostAddr. ## This accesses the 'host' field and is not a recursive call to - ## ``host=`` because the builtin dot access is preferred if it is + ## `host=` because the builtin dot access is preferred if it is ## available: s.host = value proc host*(s: Socket): int {.inline.} = ## getter of hostAddr ## This accesses the 'host' field and is not a recursive call to - ## ``host`` because the builtin dot access is preferred if it is + ## `host` because the builtin dot access is preferred if it is ## available: s.host + ``` -.. code-block:: nim + ```nim # module B import asocket var s: Socket new s s.host = 34 # same as `host=`(s, 34) + ``` -A proc defined as ``f=`` (with the trailing ``=``) is called +A proc defined as `f=` (with the trailing `=`) is called a `setter`:idx:. A setter can be called explicitly via the common backticks notation: -.. code-block:: nim - + ```nim proc `f=`(x: MyObject; value: string) = discard `f=`(myObject, "value") + ``` -``f=`` can be called implicitly in the pattern -``x.f = value`` if and only if the type of ``x`` does not have a field -named ``f`` or if ``f`` is not visible in the current module. These rules +`f=` can be called implicitly in the pattern +`x.f = value` if and only if the type of `x` does not have a field +named `f` or if `f` is not visible in the current module. These rules ensure that object fields and accessors can have the same name. Within the -module ``x.f`` is then always interpreted as field access and outside the +module `x.f` is then always interpreted as field access and outside the module it is interpreted as an accessor proc call. Command invocation syntax ------------------------- -Routines can be invoked without the ``()`` if the call is syntactically +Routines can be invoked without the `()` if the call is syntactically a statement. This command invocation syntax also works for expressions, but then only a single argument may follow. This restriction -means ``echo f 1, f 2`` is parsed as ``echo(f(1), f(2))`` and not as -``echo(f(1, f(2)))``. The method call syntax may be used to provide one +means `echo f 1, f 2` is parsed as `echo(f(1), f(2))` and not as +`echo(f(1, f(2)))`. The method call syntax may be used to provide one more argument in this case: -.. code-block:: nim + ```nim proc optarg(x: int, y: int = 0): int = x + y proc singlearg(x: int): int = 20*x @@ -3538,11 +4098,12 @@ more argument in this case: let x = optarg(1, optarg 8) # traditional procedure call with 2 arguments let y = 1.optarg optarg 8 # same thing as above, w/o the parenthesis assert x == y + ``` The command invocation syntax also can't have complex expressions as arguments. -For example: (`anonymous procs <#procedures-anonymous-procs>`_), ``if``, -``case`` or ``try``. Function calls with no arguments still needs () to -distinguish between a call and the function itself as a first class value. +For example: [anonymous procedures], `if`, +`case` or `try`. Function calls with no arguments still need () to +distinguish between a call and the function itself as a first-class value. Closures @@ -3557,71 +4118,166 @@ the closure and its enclosing scope (i.e. any modifications made to them are visible in both places). The closure environment may be allocated on the heap or on the stack if the compiler determines that this would be safe. -Creating closures in loops -~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Creating closures in loops Since closures capture local variables by reference it is often not wanted -behavior inside loop bodies. See `closureScope -<system.html#closureScope.t,untyped>`_ and `capture -<sugar.html#capture.m,openArray[typed],untyped>`_ for details on how to change this behavior. +behavior inside loop bodies. See [closureScope]( +system.html#closureScope.t,untyped) and [capture]( +sugar.html#capture.m,varargs[typed],untyped) for details on how to change this behavior. -Anonymous Procs ---------------- +Anonymous procedures +-------------------- Unnamed procedures can be used as lambda expressions to pass into other procedures: -.. code-block:: nim + ```nim var cities = @["Frankfurt", "Tokyo", "New York", "Kyiv"] - cities.sort(proc (x,y: string): int = - cmp(x.len, y.len)) + cities.sort(proc (x, y: string): int = + cmp(x.len, y.len)) + ``` -Procs as expressions can appear both as nested procs and inside top level -executable code. The `sugar <sugar.html>`_ module contains the `=>` macro +Procs as expressions can appear both as nested procs and inside top-level +executable code. The [sugar](sugar.html) module contains the `=>` macro which enables a more succinct syntax for anonymous procedures resembling lambdas as they are in languages like JavaScript, C#, etc. +Do notation +----------- + +As a special convenience notation that keeps most elements of a +regular proc expression, the `do` keyword can be used to pass +anonymous procedures to routines: + + ```nim + var cities = @["Frankfurt", "Tokyo", "New York", "Kyiv"] + + sort(cities) do (x, y: string) -> int: + cmp(x.len, y.len) + + # Less parentheses using the method plus command syntax: + cities = cities.map do (x: string) -> string: + "City of " & x + ``` + +`do` is written after the parentheses enclosing the regular proc parameters. +The proc expression represented by the `do` block is appended to the routine +call as the last argument. In calls using the command syntax, the `do` block +will bind to the immediately preceding expression rather than the command call. + +`do` with a parameter list or pragma list corresponds to an anonymous `proc`, +however `do` without parameters or pragmas is treated as a normal statement +list. This allows macros to receive both indented statement lists as an +argument in inline calls, as well as a direct mirror of Nim's routine syntax. + + ```nim + # Passing a statement list to an inline macro: + macroResults.add quote do: + if not `ex`: + echo `info`, ": Check failed: ", `expString` + + # Processing a routine definition in a macro: + rpc(router, "add") do (a, b: int) -> int: + result = a + b + ``` Func ---- -The ``func`` keyword introduces a shortcut for a `noSideEffect`:idx: proc. +The `func` keyword introduces a shortcut for a `noSideEffect`:idx: proc. -.. code-block:: nim + ```nim func binarySearch[T](a: openArray[T]; elem: T): int + ``` Is short for: -.. code-block:: nim + ```nim proc binarySearch[T](a: openArray[T]; elem: T): int {.noSideEffect.} + ``` + + + +Routines +-------- + +A routine is a symbol of kind: `proc`, `func`, `method`, `iterator`, `macro`, `template`, `converter`. +Type bound operators +-------------------- +A type bound operator is a `proc` or `func` whose name starts with `=` but isn't an operator +(i.e. containing only symbols, such as `==`). These are unrelated to setters +(see [Properties]), which instead end in `=`. +A type bound operator declared for a type applies to the type regardless of whether +the operator is in scope (including if it is private). + + ```nim + # foo.nim: + var witness* = 0 + type Foo[T] = object + proc initFoo*(T: typedesc): Foo[T] = discard + proc `=destroy`[T](x: var Foo[T]) = witness.inc # type bound operator + + # main.nim: + import foo + block: + var a = initFoo(int) + doAssert witness == 0 + doAssert witness == 1 + block: + var a = initFoo(int) + doAssert witness == 1 + `=destroy`(a) # can be called explicitly, even without being in scope + doAssert witness == 2 + # will still be called upon exiting scope + doAssert witness == 3 + ``` + +Type bound operators are: +`=destroy`, `=copy`, `=sink`, `=trace`, `=deepcopy`, `=wasMoved`, `=dup`. + +These operations can be *overridden* instead of *overloaded*. This means that +the implementation is automatically lifted to structured types. For instance, +if the type `T` has an overridden assignment operator `=`, this operator is +also used for assignments of the type `seq[T]`. + +Since these operations are bound to a type, they have to be bound to a +nominal type for reasons of simplicity of implementation; this means an +overridden `deepCopy` for `ref T` is really bound to `T` and not to `ref T`. +This also means that one cannot override `deepCopy` for both `ptr T` and +`ref T` at the same time, instead a distinct or object helper type has to be +used for one pointer type. + +For more details on some of those procs, see +[Lifetime-tracking hooks](destructors.html#lifetimeminustracking-hooks). Nonoverloadable builtins ------------------------ -The following builtin procs cannot be overloaded for reasons of implementation -simplicity (they require specialized semantic checking):: +The following built-in procs cannot be overloaded for reasons of implementation +simplicity (they require specialized semantic checking): - declared, defined, definedInScope, compiles, sizeof, - is, shallowCopy, getAst, astToStr, spawn, procCall + declared, defined, definedInScope, compiles, sizeof, + is, shallowCopy, getAst, astToStr, spawn, procCall -Thus they act more like keywords than like ordinary identifiers; unlike a +Thus, they act more like keywords than like ordinary identifiers; unlike a keyword however, a redefinition may `shadow`:idx: the definition in -the ``system`` module. From this list the following should not be written in dot -notation ``x.f`` since ``x`` cannot be type checked before it gets passed -to ``f``:: +the [system](system.html) module. +From this list the following should not be written in dot +notation `x.f` since `x` cannot be type-checked before it gets passed +to `f`: - declared, defined, definedInScope, compiles, getAst, astToStr + declared, defined, definedInScope, compiles, getAst, astToStr Var parameters -------------- -The type of a parameter may be prefixed with the ``var`` keyword: +The type of a parameter may be prefixed with the `var` keyword: -.. code-block:: nim + ```nim proc divmod(a, b: int; res, remainder: var int) = res = a div b remainder = a mod b @@ -3632,14 +4288,15 @@ The type of a parameter may be prefixed with the ``var`` keyword: divmod(8, 5, x, y) # modifies x and y assert x == 1 assert y == 3 + ``` -In the example, ``res`` and ``remainder`` are `var parameters`. +In the example, `res` and `remainder` are `var parameters`. Var parameters can be modified by the procedure and the changes are visible to the caller. The argument passed to a var parameter has to be an l-value. Var parameters are implemented as hidden pointers. The above example is equivalent to: -.. code-block:: nim + ```nim proc divmod(a, b: int; res, remainder: ptr int) = res[] = a div b remainder[] = a mod b @@ -3649,11 +4306,12 @@ above example is equivalent to: divmod(8, 5, addr(x), addr(y)) assert x == 1 assert y == 3 + ``` In the examples, var parameters or pointers are used to provide two return values. This can be done in a cleaner way by returning a tuple: -.. code-block:: nim + ```nim proc divmod(a, b: int): tuple[res, remainder: int] = (a div b, a mod b) @@ -3661,16 +4319,18 @@ return values. This can be done in a cleaner way by returning a tuple: assert t.res == 1 assert t.remainder == 3 + ``` One can use `tuple unpacking`:idx: to access the tuple's fields: -.. code-block:: nim + ```nim var (x, y) = divmod(8, 5) # tuple unpacking assert x == 1 assert y == 3 + ``` -**Note**: ``var`` parameters are never necessary for efficient parameter +**Note**: `var` parameters are never necessary for efficient parameter passing. Since non-var parameters cannot be modified the compiler is always free to pass arguments by reference if it considers it can speed up execution. @@ -3678,10 +4338,10 @@ free to pass arguments by reference if it considers it can speed up execution. Var return type --------------- -A proc, converter or iterator may return a ``var`` type which means that the +A proc, converter, or iterator may return a `var` type which means that the returned value is an l-value and can be modified by the caller: -.. code-block:: nim + ```nim var g = 0 proc writeAccessToG(): var int = @@ -3689,41 +4349,44 @@ returned value is an l-value and can be modified by the caller: writeAccessToG() = 6 assert g == 6 + ``` It is a static error if the implicitly introduced pointer could be used to access a location beyond its lifetime: -.. code-block:: nim + ```nim proc writeAccessToG(): var int = var g = 0 result = g # Error! + ``` -For iterators, a component of a tuple return type can have a ``var`` type too: +For iterators, a component of a tuple return type can have a `var` type too: -.. code-block:: nim + ```nim iterator mpairs(a: var seq[string]): tuple[key: int, val: var string] = for i in 0..a.high: yield (i, a[i]) + ``` -In the standard library every name of a routine that returns a ``var`` type -starts with the prefix ``m`` per convention. +In the standard library every name of a routine that returns a `var` type +starts with the prefix `m` per convention. -.. include:: manual/var_t_return.rst +.. include:: manual/var_t_return.md -Future directions -~~~~~~~~~~~~~~~~~ +### Future directions Later versions of Nim can be more precise about the borrowing rule with a syntax like: -.. code-block:: nim + ```nim proc foo(other: Y; container: var X): var T from container + ``` -Here ``var T from container`` explicitly exposes that the +Here `var T from container` explicitly exposes that the location is derived from the second parameter (called -'container' in this case). The syntax ``var T from p`` specifies a type -``varTy[T, 2]`` which is incompatible with ``varTy[T, 1]``. +'container' in this case). The syntax `var T from p` specifies a type +`varTy[T, 2]` which is incompatible with `varTy[T, 1]`. NRVO @@ -3736,15 +4399,15 @@ See https://github.com/nim-lang/RFCs/issues/230 for more information. The return value is represented inside the body of a routine as the special `result`:idx: variable. This allows for a mechanism much like C++'s "named return value optimization" (`NRVO`:idx:). NRVO means that the stores -to ``result`` inside ``p`` directly affect the destination ``dest`` -in ``let/var dest = p(args)`` (definition of ``dest``) and also in ``dest = p(args)`` -(assignment to ``dest``). This is achieved by rewriting ``dest = p(args)`` -to ``p'(args, dest)`` where ``p'`` is a variation of ``p`` that returns ``void`` and -receives a hidden mutable parameter representing ``result``. +to `result` inside `p` directly affect the destination `dest` +in `let/var dest = p(args)` (definition of `dest`) and also in `dest = p(args)` +(assignment to `dest`). This is achieved by rewriting `dest = p(args)` +to `p'(args, dest)` where `p'` is a variation of `p` that returns `void` and +receives a hidden mutable parameter representing `result`. Informally: -.. code-block:: nim + ```nim proc p(): BigT = ... var x = p() @@ -3756,17 +4419,17 @@ Informally: var x; p(x) p(x) + ``` -Let ``T``'s be ``p``'s return type. NRVO applies for ``T`` -if ``sizeof(T) >= N`` (where ``N`` is implementation dependent), +Let `T`'s be `p`'s return type. NRVO applies for `T` +if `sizeof(T) >= N` (where `N` is implementation dependent), in other words, it applies for "big" structures. -If ``p`` can raise an exception, NRVO applies regardless. This can produce +If `p` can raise an exception, NRVO applies regardless. This can produce observable differences in behavior: -.. code-block:: nim - + ```nim type BigT = array[16, int] @@ -3783,36 +4446,69 @@ observable differences in behavior: doAssert x == [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0] main() + ``` -However, the current implementation produces a warning in these cases. -There are different ways to deal with this warning: - -1. Disable the warning via ``{.push warning[ObservableStores]: off.}`` ... ``{.pop.}``. - Then one may need to ensure that ``p`` only raises *before* any stores to ``result`` - happen. - -2. One can use a temporary helper variable, for example instead of ``x = p(8)`` - use ``let tmp = p(8); x = tmp``. +The compiler can produce a warning in these cases, however this behavior is +turned off by default. It can be enabled for a section of code via the +`warning[ObservableStores]` and `push`/`pop` pragmas. Take the above code +as an example: + ```nim + {.push warning[ObservableStores]: on.} + main() + {.pop.} + ``` Overloading of the subscript operator ------------------------------------- -The ``[]`` subscript operator for arrays/openarrays/sequences can be overloaded. - - -Multi-methods +The `[]` subscript operator for arrays/openarrays/sequences can be overloaded +for any type (with some exceptions) by defining a routine with the name `[]`. + + ```nim + type Foo = object + data: seq[int] + + proc `[]`(foo: Foo, i: int): int = + result = foo.data[i] + + let foo = Foo(data: @[1, 2, 3]) + echo foo[1] # 2 + ``` + +Assignment to subscripts can also be overloaded by naming a routine `[]=`, +which has precedence over assigning to the result of `[]`. + + ```nim + type Foo = object + data: seq[int] + + proc `[]`(foo: Foo, i: int): int = + result = foo.data[i] + proc `[]=`(foo: var Foo, i: int, val: int) = + foo.data[i] = val + + var foo = Foo(data: @[1, 2, 3]) + echo foo[1] # 2 + foo[1] = 5 + echo foo.data # @[1, 5, 3] + echo foo[1] # 5 + ``` + +Overloads of the subscript operator cannot be applied to routine or type +symbols themselves, as this conflicts with the syntax for instantiating +generic parameters, i.e. `foo[int](1, 2, 3)` or `Foo[int]`. + + +Methods ============= -**Note:** Starting from Nim 0.20, to use multi-methods one must explicitly pass -``--multimethods:on`` when compiling. - -Procedures always use static dispatch. Multi-methods use dynamic +Procedures always use static dispatch. Methods use dynamic dispatch. For dynamic dispatch to work on an object it should be a reference type. -.. code-block:: nim + ```nim type Expression = ref object of RootObj ## abstract base class for an expression Literal = ref object of Expression @@ -3840,21 +4536,51 @@ type. result.b = b echo eval(newPlus(newPlus(newLit(1), newLit(2)), newLit(4))) + ``` -In the example the constructors ``newLit`` and ``newPlus`` are procs -because they should use static binding, but ``eval`` is a method because it +In the example the constructors `newLit` and `newPlus` are procs +because they should use static binding, but `eval` is a method because it requires dynamic binding. As can be seen in the example, base methods have to be annotated with -the `base`:idx: pragma. The ``base`` pragma also acts as a reminder for the -programmer that a base method ``m`` is used as the foundation to determine all -the effects that a call to ``m`` might cause. +the `base`:idx: pragma. The `base` pragma also acts as a reminder for the +programmer that a base method `m` is used as the foundation to determine all +the effects that a call to `m` might cause. **Note**: Compile-time execution is not (yet) supported for methods. **Note**: Starting from Nim 0.20, generic methods are deprecated. +Multi-methods +-------------- + +**Note:** Starting from Nim 0.20, to use multi-methods one must explicitly pass +`--multimethods:on`:option: when compiling. + +In a multi-method, all parameters that have an object type are used for the +dispatching: + + ```nim test = "nim c --multiMethods:on $1" + type + Thing = ref object of RootObj + Unit = ref object of Thing + x: int + + method collide(a, b: Thing) {.base, inline.} = + quit "to override!" + + method collide(a: Thing, b: Unit) {.inline.} = + echo "1" + + method collide(a: Unit, b: Thing) {.inline.} = + echo "2" + + var a, b: Unit + new a + new b + collide(a, b) # output: 2 + ``` Inhibit dynamic method resolution via procCall ----------------------------------------------- @@ -3863,9 +4589,7 @@ Dynamic method resolution can be inhibited via the builtin `system.procCall`:idx This is somewhat comparable to the `super`:idx: keyword that traditional OOP languages offer. -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" type Thing = ref object of RootObj Unit = ref object of Thing @@ -3878,29 +4602,30 @@ languages offer. # Call the base method: procCall m(Thing(a)) echo "1" + ``` Iterators and the for statement =============================== The `for`:idx: statement is an abstract mechanism to iterate over the elements -of a container. It relies on an `iterator`:idx: to do so. Like ``while`` -statements, ``for`` statements open an `implicit block`:idx:, so that they -can be left with a ``break`` statement. +of a container. It relies on an `iterator`:idx: to do so. Like `while` +statements, `for` statements open an `implicit block`:idx: so that they +can be left with a `break` statement. -The ``for`` loop declares iteration variables - their scope reaches until the +The `for` loop declares iteration variables - their scope reaches until the end of the loop body. The iteration variables' types are inferred by the return type of the iterator. An iterator is similar to a procedure, except that it can be called in the -context of a ``for`` loop. Iterators provide a way to specify the iteration over -an abstract type. A key role in the execution of a ``for`` loop plays the -``yield`` statement in the called iterator. Whenever a ``yield`` statement is -reached the data is bound to the ``for`` loop variables and control continues -in the body of the ``for`` loop. The iterator's local variables and execution +context of a `for` loop. Iterators provide a way to specify the iteration over +an abstract type. The `yield` statement in the called iterator plays a key +role in the execution of a `for` loop. Whenever a `yield` statement is +reached, the data is bound to the `for` loop variables and control continues +in the body of the `for` loop. The iterator's local variables and execution state are automatically saved between calls. Example: -.. code-block:: nim + ```nim # this definition exists in the system module iterator items*(a: string): char {.inline.} = var i = 0 @@ -3910,15 +4635,17 @@ state are automatically saved between calls. Example: for ch in items("hello world"): # `ch` is an iteration variable echo ch + ``` -The compiler generates code as if the programmer would have written this: +The compiler generates code as if the programmer had written this: -.. code-block:: nim + ```nim var i = 0 while i < len(a): var ch = a[i] echo ch inc(i) + ``` If the iterator yields a tuple, there can be as many iteration variables as there are components in the tuple. The i'th iteration variable's type is @@ -3926,24 +4653,25 @@ the type of the i'th component. In other words, implicit tuple unpacking in a for loop context is supported. Implicit items/pairs invocations -------------------------------- +-------------------------------- -If the for loop expression ``e`` does not denote an iterator and the for loop -has exactly 1 variable, the for loop expression is rewritten to ``items(e)``; -ie. an ``items`` iterator is implicitly invoked: +If the for loop expression `e` does not denote an iterator and the for loop +has exactly 1 variable, the for loop expression is rewritten to `items(e)`; +i.e. an `items` iterator is implicitly invoked: -.. code-block:: nim + ```nim for x in [1,2,3]: echo x + ``` -If the for loop has exactly 2 variables, a ``pairs`` iterator is implicitly +If the for loop has exactly 2 variables, a `pairs` iterator is implicitly invoked. -Symbol lookup of the identifiers ``items``/``pairs`` is performed after -the rewriting step, so that all overloads of ``items``/``pairs`` are taken +Symbol lookup of the identifiers `items`/`pairs` is performed after +the rewriting step, so that all overloads of `items`/`pairs` are taken into account. -First class iterators +First-class iterators --------------------- There are 2 kinds of iterators in Nim: *inline* and *closure* iterators. @@ -3952,17 +4680,17 @@ leading to zero overhead for the abstraction, but may result in a heavy increase in code size. Caution: the body of a for loop over an inline iterator is inlined into -each ``yield`` statement appearing in the iterator code, +each `yield` statement appearing in the iterator code, so ideally the code should be refactored to contain a single yield when possible to avoid code bloat. Inline iterators are second class citizens; They can be passed as parameters only to other inlining code facilities like -templates, macros and other inline iterators. +templates, macros, and other inline iterators. In contrast to that, a `closure iterator`:idx: can be passed around more freely: -.. code-block:: nim + ```nim iterator count0(): int {.closure.} = yield 0 @@ -3977,25 +4705,25 @@ In contrast to that, a `closure iterator`:idx: can be passed around more freely: invoke(count0) invoke(count2) + ``` Closure iterators and inline iterators have some restrictions: 1. For now, a closure iterator cannot be executed at compile time. -2. ``return`` is allowed in a closure iterator but not in an inline iterator +2. `return` is allowed in a closure iterator but not in an inline iterator (but rarely useful) and ends the iteration. -3. Neither inline nor closure iterators can be recursive. -4. Neither inline nor closure iterators have the special ``result`` variable. -5. Closure iterators are not supported by the js backend. +3. Inline iterators cannot be recursive. +4. Neither inline nor closure iterators have the special `result` variable. -Iterators that are neither marked ``{.closure.}`` nor ``{.inline.}`` explicitly +Iterators that are neither marked `{.closure.}` nor `{.inline.}` explicitly default to being inline, but this may change in future versions of the implementation. -The ``iterator`` type is always of the calling convention ``closure`` +The `iterator` type is always of the calling convention `closure` implicitly; the following example shows how to use iterators to implement a `collaborative tasking`:idx: system: -.. code-block:: nim + ```nim # simple tasking: type Task = iterator (ticker: int) @@ -4025,15 +4753,16 @@ a `collaborative tasking`:idx: system: inc ticker runTasks(a1, a2) + ``` -The builtin ``system.finished`` can be used to determine if an iterator has +The builtin `system.finished` can be used to determine if an iterator has finished its operation; no exception is raised on an attempt to invoke an iterator that has already finished its work. -Note that ``system.finished`` is error prone to use because it only returns -``true`` one iteration after the iterator has finished: +Note that `system.finished` is error-prone to use because it only returns +`true` one iteration after the iterator has finished: -.. code-block:: nim + ```nim iterator mycount(a, b: int): int {.closure.} = var x = a while x <= b: @@ -4049,18 +4778,20 @@ Note that ``system.finished`` is error prone to use because it only returns 2 3 0 + ``` -Instead this code has to be used: +Instead, this code has to be used: -.. code-block:: nim + ```nim var c = mycount # instantiate the iterator while true: let value = c(1, 3) if finished(c): break # and discard 'value'! echo value + ``` It helps to think that the iterator actually returns a -pair ``(value, done)`` and ``finished`` is used to access the hidden ``done`` +pair `(value, done)` and `finished` is used to access the hidden `done` field. @@ -4068,7 +4799,7 @@ Closure iterators are *resumable functions* and so one has to provide the arguments to every call. To get around this limitation one can capture parameters of an outer factory proc: -.. code-block:: nim + ```nim proc mycount(a, b: int): iterator (): int = result = iterator (): int = var x = a @@ -4080,26 +4811,67 @@ parameters of an outer factory proc: for f in foo(): echo f + ``` + +The call can be made more like an inline iterator with a for loop macro: + + ```nim + import std/macros + macro toItr(x: ForLoopStmt): untyped = + let expr = x[0] + let call = x[1][1] # Get foo out of toItr(foo) + let body = x[2] + result = quote do: + block: + let itr = `call` + for `expr` in itr(): + `body` + + for f in toItr(mycount(1, 4)): # using early `proc mycount` + echo f + ``` + +Because of full backend function call apparatus involvement, closure iterator +invocation is typically higher cost than inline iterators. Adornment by +a macro wrapper at the call site like this is a possibly useful reminder. +The factory `proc`, as an ordinary procedure, can be recursive. The +above macro allows such recursion to look much like a recursive iterator +would. For example: + ```nim + proc recCountDown(n: int): iterator(): int = + result = iterator(): int = + if n > 0: + yield n + for e in toItr(recCountDown(n - 1)): + yield e + + for i in toItr(recCountDown(6)): # Emits: 6 5 4 3 2 1 + echo i + ``` + + +See also [iterable] for passing iterators to templates and macros. Converters ========== A converter is like an ordinary proc except that it enhances -the "implicitly convertible" type relation (see `Convertible relation`_): +the "implicitly convertible" type relation (see [Convertible relation]): -.. code-block:: nim + ```nim # bad style ahead: Nim is not C. converter toBool(x: int): bool = x != 0 if 4: echo "compiles" + ``` A converter can also be explicitly invoked for improved readability. Note that implicit converter chaining is not supported: If there is a converter from -type A to type B and from type B to type C the implicit conversion from A to C +type A to type B and from type B to type C, the implicit conversion from A to C is not provided. @@ -4108,7 +4880,7 @@ Type sections Example: -.. code-block:: nim + ```nim type # example demonstrating mutually recursive types Node = ref object # an object managed by the garbage collector (ref) le, ri: Node # left and right subtrees @@ -4118,12 +4890,13 @@ Example: name: string # the symbol's name line: int # the line the symbol was declared in code: Node # the symbol's abstract syntax tree + ``` -A type section begins with the ``type`` keyword. It contains multiple +A type section begins with the `type` keyword. It contains multiple type definitions. A type definition binds a type to a name. Type definitions can be recursive or even mutually recursive. Mutually recursive types are only -possible within a single ``type`` section. Nominal types like ``objects`` -or ``enums`` can only be defined in a ``type`` section. +possible within a single `type` section. Nominal types like `objects` +or `enums` can only be defined in a `type` section. @@ -4135,7 +4908,7 @@ Try statement Example: -.. code-block:: nim + ```nim # read the first two lines of a text file that should contain numbers # and tries to add them var @@ -4147,139 +4920,143 @@ Example: echo "sum: " & $(parseInt(a) + parseInt(b)) except OverflowDefect: echo "overflow!" - except ValueError: - echo "could not convert string to integer" - except IOError: - echo "IO error!" - except: - echo "Unknown exception!" + except ValueError, IOError: + echo "catch multiple exceptions!" + except CatchableError: + echo "Catchable exception!" finally: close(f) + ``` -The statements after the ``try`` are executed in sequential order unless -an exception ``e`` is raised. If the exception type of ``e`` matches any -listed in an ``except`` clause the corresponding statements are executed. -The statements following the ``except`` clauses are called +The statements after the `try` are executed in sequential order unless +an exception `e` is raised. If the exception type of `e` matches any +listed in an `except` clause, the corresponding statements are executed. +The statements following the `except` clauses are called `exception handlers`:idx:. -The empty `except`:idx: clause is executed if there is an exception that is -not listed otherwise. It is similar to an ``else`` clause in ``if`` statements. - If there is a `finally`:idx: clause, it is always executed after the exception handlers. The exception is *consumed* in an exception handler. However, an exception handler may raise another exception. If the exception is not handled, it is propagated through the call stack. This means that often -the rest of the procedure - that is not within a ``finally`` clause - +the rest of the procedure - that is not within a `finally` clause - is not executed (if an exception occurs). Try expression -------------- -Try can also be used as an expression; the type of the ``try`` branch then -needs to fit the types of ``except`` branches, but the type of the ``finally`` -branch always has to be ``void``: +Try can also be used as an expression; the type of the `try` branch then +needs to fit the types of `except` branches, but the type of the `finally` +branch always has to be `void`: -.. code-block:: nim - from strutils import parseInt + ```nim test + from std/strutils import parseInt let x = try: parseInt("133a") - except: -1 + except ValueError: -1 finally: echo "hi" + ``` -To prevent confusing code there is a parsing limitation; if the ``try`` -follows a ``(`` it has to be written as a one liner: +To prevent confusing code there is a parsing limitation; if the `try` +follows a `(` it has to be written as a one liner: -.. code-block:: nim - let x = (try: parseInt("133a") except: -1) + ```nim test + from std/strutils import parseInt + let x = (try: parseInt("133a") except ValueError: -1) + ``` Except clauses -------------- -Within an ``except`` clause it is possible to access the current exception +Within an `except` clause it is possible to access the current exception using the following syntax: -.. code-block:: nim + ```nim try: # ... except IOError as e: # Now use "e" echo "I/O error: " & e.msg + ``` -Alternatively, it is possible to use ``getCurrentException`` to retrieve the +Alternatively, it is possible to use `getCurrentException` to retrieve the exception that has been raised: -.. code-block:: nim + ```nim try: # ... except IOError: let e = getCurrentException() # Now use "e" + ``` -Note that ``getCurrentException`` always returns a ``ref Exception`` +Note that `getCurrentException` always returns a `ref Exception` type. If a variable of the proper type is needed (in the example -above, ``IOError``), one must convert it explicitly: +above, `IOError`), one must convert it explicitly: -.. code-block:: nim + ```nim try: # ... except IOError: let e = (ref IOError)(getCurrentException()) # "e" is now of the proper type + ``` However, this is seldom needed. The most common case is to extract an -error message from ``e``, and for such situations it is enough to use -``getCurrentExceptionMsg``: +error message from `e`, and for such situations, it is enough to use +`getCurrentExceptionMsg`: -.. code-block:: nim + ```nim try: # ... - except: + except CatchableError: echo getCurrentExceptionMsg() + ``` Custom exceptions ----------------- -Is it possible to create custom exceptions. A custom exception is a custom type: +It is possible to create custom exceptions. A custom exception is a custom type: -.. code-block:: nim + ```nim type LoadError* = object of Exception + ``` -Ending the custom exception's name with ``Error`` is recommended. +Ending the custom exception's name with `Error` is recommended. -Custom exceptions can be raised like any others, e.g.: +Custom exceptions can be raised just like any other exception, e.g.: -.. code-block:: nim + ```nim raise newException(LoadError, "Failed to load data") + ``` Defer statement --------------- -Instead of a ``try finally`` statement a ``defer`` statement can be used. - -Any statements following the ``defer`` in the current block will be considered -to be in an implicit try block: +Instead of a `try finally` statement a `defer` statement can be used, which +avoids lexical nesting and offers more flexibility in terms of scoping as shown +below. -.. code-block:: nim - :test: "nim c $1" +Any statements following the `defer` will be considered +to be in an implicit try block in the current block: + ```nim test = "nim c $1" proc main = - var f = open("numbers.txt") + var f = open("numbers.txt", fmWrite) defer: close(f) f.write "abc" f.write "def" + ``` Is rewritten to: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" proc main = var f = open("numbers.txt") try: @@ -4287,8 +5064,35 @@ Is rewritten to: f.write "def" finally: close(f) + ``` + +When `defer` is at the outermost scope of a template/macro, its scope extends +to the block where the template/macro is called from: -Top level ``defer`` statements are not supported + ```nim test = "nim c $1" + template safeOpenDefer(f, path) = + var f = open(path, fmWrite) + defer: close(f) + + template safeOpenFinally(f, path, body) = + var f = open(path, fmWrite) + try: body # without `defer`, `body` must be specified as parameter + finally: close(f) + + block: + safeOpenDefer(f, "/tmp/z01.txt") + f.write "abc" + block: + safeOpenFinally(f, "/tmp/z01.txt"): + f.write "abc" # adds a lexical scope + block: + var f = open("/tmp/z01.txt", fmWrite) + try: + f.write "abc" # adds a lexical scope + finally: close(f) + ``` + +Top-level `defer` statements are not supported since it's unclear what such a statement should refer to. @@ -4297,33 +5101,66 @@ Raise statement Example: -.. code-block:: nim + ```nim raise newException(IOError, "IO failed") + ``` Apart from built-in operations like array indexing, memory allocation, etc. -the ``raise`` statement is the only way to raise an exception. +the `raise` statement is the only way to raise an exception. .. XXX document this better! If no exception name is given, the current exception is `re-raised`:idx:. The `ReraiseDefect`:idx: exception is raised if there is no exception to -re-raise. It follows that the ``raise`` statement *always* raises an +re-raise. It follows that the `raise` statement *always* raises an exception. Exception hierarchy ------------------- -The exception tree is defined in the `system <system.html>`_ module. -Every exception inherits from ``system.Exception``. Exceptions that indicate -programming bugs inherit from ``system.Defect`` (which is a subtype of ``Exception``) +The exception tree is defined in the [system](system.html) module. +Every exception inherits from `system.Exception`. Exceptions that indicate +programming bugs inherit from `system.Defect` (which is a subtype of `Exception`) and are strictly speaking not catchable as they can also be mapped to an operation that terminates the whole process. If panics are turned into exceptions, these exceptions inherit from `Defect`. Exceptions that indicate any other runtime error that can be caught inherit from -``system.CatchableError`` (which is a subtype of ``Exception``). - +`system.CatchableError` (which is a subtype of `Exception`). + +``` +Exception +|-- CatchableError +| |-- IOError +| | `-- EOFError +| |-- OSError +| |-- ResourceExhaustedError +| `-- ValueError +| `-- KeyError +`-- Defect + |-- AccessViolationDefect + |-- ArithmeticDefect + | |-- DivByZeroDefect + | `-- OverflowDefect + |-- AssertionDefect + |-- DeadThreadDefect + |-- FieldDefect + |-- FloatingPointDefect + | |-- FloatDivByZeroDefect + | |-- FloatInvalidOpDefect + | |-- FloatOverflowDefect + | |-- FloatUnderflowDefect + | `-- InexactDefect + |-- IndexDefect + |-- NilAccessDefect + |-- ObjectAssignmentDefect + |-- ObjectConversionDefect + |-- OutOfMemoryDefect + |-- RangeDefect + |-- ReraiseDefect + `-- StackOverflowDefect +``` Imported exceptions ------------------- @@ -4332,9 +5169,7 @@ It is possible to raise/catch imported C++ exceptions. Types imported using `importcpp` can be raised or caught. Exceptions are raised by value and caught by reference. Example: -.. code-block:: nim - :test: "nim cpp -r $1" - + ```nim test = "nim cpp -r $1" type CStdException {.importcpp: "std::exception", header: "<exception>", inheritable.} = object ## does not inherit from `RootObj`, so we use `inheritable` instead @@ -4365,6 +5200,7 @@ caught by reference. Example: doAssert $b == "foo3" fn() + ``` **Note:** `getCurrentException()` and `getCurrentExceptionMsg()` are not available for imported exceptions from C++. One needs to use the `except ImportedException as x:` syntax @@ -4374,6 +5210,10 @@ and rely on functionality of the `x` object to get exception details. Effect system ============= +**Note**: The rules for effect tracking changed with the release of version +1.6 of the Nim compiler. + + Exception tracking ------------------ @@ -4381,31 +5221,28 @@ Nim supports exception tracking. The `raises`:idx: pragma can be used to explicitly define which exceptions a proc/iterator/method/converter is allowed to raise. The compiler verifies this: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" proc p(what: bool) {.raises: [IOError, OSError].} = if what: raise newException(IOError, "IO") else: raise newException(OSError, "OS") + ``` -An empty ``raises`` list (``raises: []``) means that no exception may be raised: +An empty `raises` list (`raises: []`) means that no exception may be raised: -.. code-block:: nim + ```nim proc p(): bool {.raises: [].} = try: unsafeCall() result = true - except: + except CatchableError: result = false + ``` -A ``raises`` list can also be attached to a proc type. This affects type +A `raises` list can also be attached to a proc type. This affects type compatibility: -.. code-block:: nim - :test: "nim c $1" - :status: 1 - + ```nim test = "nim c $1" status = 1 type Callback = proc (s: string) {.raises: [IOError].} var @@ -4415,112 +5252,271 @@ compatibility: raise newException(OSError, "OS") c = p # type error + ``` + + +For a routine `p`, the compiler uses inference rules to determine the set of +possibly raised exceptions; the algorithm operates on `p`'s call graph: + +1. Every indirect call via some proc type `T` is assumed to + raise `system.Exception` (the base type of the exception hierarchy) and + thus any exception unless `T` has an explicit `raises` list. + However, if the call is of the form `f(...)` where `f` is a parameter of + the currently analyzed routine that is marked as `.effectsOf: f`, it is ignored. + The call is optimistically assumed to have no effect. + Rule 2 compensates for this case. +2. Every expression `e` of some proc type within a call that is passed to parameter + marked as `.effectsOf` of proc `p` is assumed to be called indirectly and thus + its raises list is added to `p`'s raises list. +3. Every call to a proc `q` which has an unknown body (due to a forward + declaration) is assumed to + raise `system.Exception` unless `q` has an explicit `raises` list. + Procs that are `importc`'ed are assumed to have `.raises: []`, unless explicitly + declared otherwise. +4. Every call to a method `m` is assumed to + raise `system.Exception` unless `m` has an explicit `raises` list. +5. For every other call, the analysis can determine an exact `raises` list. +6. For determining a `raises` list, the `raise` and `try` statements + of `p` are taken into consideration. + + +Exceptions inheriting from `system.Defect` are not tracked with +the `.raises: []` exception tracking mechanism. This is more consistent with the +built-in operations. The following code is valid: + + ```nim + proc mydiv(a, b): int {.raises: [].} = + a div b # can raise an DivByZeroDefect + ``` + +And so is: + + ```nim + proc mydiv(a, b): int {.raises: [].} = + if b == 0: raise newException(DivByZeroDefect, "division by zero") + else: result = a div b + ``` -For a routine ``p`` the compiler uses inference rules to determine the set of -possibly raised exceptions; the algorithm operates on ``p``'s call graph: - -1. Every indirect call via some proc type ``T`` is assumed to - raise ``system.Exception`` (the base type of the exception hierarchy) and - thus any exception unless ``T`` has an explicit ``raises`` list. - However if the call is of the form ``f(...)`` where ``f`` is a parameter - of the currently analysed routine it is ignored. The call is optimistically - assumed to have no effect. Rule 2 compensates for this case. -2. Every expression of some proc type within a call that is not a call - itself (and not nil) is assumed to be called indirectly somehow and thus - its raises list is added to ``p``'s raises list. -3. Every call to a proc ``q`` which has an unknown body (due to a forward - declaration or an ``importc`` pragma) is assumed to - raise ``system.Exception`` unless ``q`` has an explicit ``raises`` list. -4. Every call to a method ``m`` is assumed to - raise ``system.Exception`` unless ``m`` has an explicit ``raises`` list. -5. For every other call the analysis can determine an exact ``raises`` list. -6. For determining a ``raises`` list, the ``raise`` and ``try`` statements - of ``p`` are taken into consideration. - -Rules 1-2 ensure the following works: - -.. code-block:: nim - proc noRaise(x: proc()) {.raises: [].} = - # unknown call that might raise anything, but valid: - x() +The reason for this is that `DivByZeroDefect` inherits from `Defect` and +with `--panics:on`:option: Defects become unrecoverable errors. +(Since version 1.4 of the language.) + + +EffectsOf annotation +-------------------- + +Rules 1-2 of the exception tracking inference rules (see the previous section) +ensure the following works: + + ```nim + proc weDontRaiseButMaybeTheCallback(callback: proc()) {.raises: [], effectsOf: callback.} = + callback() proc doRaise() {.raises: [IOError].} = raise newException(IOError, "IO") proc use() {.raises: [].} = # doesn't compile! Can raise IOError! - noRaise(doRaise) + weDontRaiseButMaybeTheCallback(doRaise) + ``` + +As can be seen from the example, a parameter of type `proc (...)` can be +annotated as `.effectsOf`. Such a parameter allows for effect polymorphism: +The proc `weDontRaiseButMaybeTheCallback` raises the exceptions +that `callback` raises. So in many cases a callback does not cause the compiler to be overly -conservative in its effect analysis. +conservative in its effect analysis: -Exceptions inheriting from ``system.Defect`` are not tracked with -the ``.raises: []`` exception tracking mechanism. This is more consistent with the -built-in operations. The following code is valid:: + ```nim test = "nim c $1" status = 1 + {.push warningAsError[Effect]: on.} -.. code-block:: nim + import std/algorithm - proc mydiv(a, b): int {.raises: [].} = - a div b # can raise an DivByZeroDefect + type + MyInt = distinct int -And so is:: + var toSort = @[MyInt 1, MyInt 2, MyInt 3] -.. code-block:: nim + proc cmpN(a, b: MyInt): int = + cmp(a.int, b.int) - proc mydiv(a, b): int {.raises: [].} = - if b == 0: raise newException(DivByZeroDefect, "division by zero") - else: result = a div b + proc harmless {.raises: [].} = + toSort.sort cmpN + proc cmpE(a, b: MyInt): int {.raises: [Exception].} = + cmp(a.int, b.int) + + proc harmful {.raises: [].} = + # does not compile, `sort` can now raise Exception + toSort.sort cmpE + ``` -The reason for this is that ``DivByZeroDefect`` inherits from ``Defect`` and -with ``--panics:on`` Defects become unrecoverable errors. -(Since version 1.4 of the language.) Tag tracking ------------ -The exception tracking is part of Nim's `effect system`:idx:. Raising an -exception is an *effect*. Other effects can also be defined. A user defined -effect is a means to *tag* a routine and to perform checks against this tag: - -.. code-block:: nim - :test: "nim c $1" - :status: 1 +Exception tracking is part of Nim's `effect system`:idx:. Raising an exception +is an *effect*. Other effects can also be defined. A user defined effect is a +means to *tag* a routine and to perform checks against this tag: + ```nim test = "nim c --warningAsError:Effect:on $1" status = 1 type IO = object ## input/output effect proc readLine(): string {.tags: [IO].} = discard - proc no_IO_please() {.tags: [].} = + proc no_effects_please() {.tags: [].} = # the compiler prevents this: let x = readLine() + ``` -A tag has to be a type name. A ``tags`` list - like a ``raises`` list - can +A tag has to be a type name. A `tags` list - like a `raises` list - can also be attached to a proc type. This affects type compatibility. The inference for tag tracking is analogous to the inference for exception tracking. +There is also a way which can be used to forbid certain effects: + + ```nim test = "nim c --warningAsError:Effect:on $1" status = 1 + type IO = object ## input/output effect + proc readLine(): string {.tags: [IO].} = discard + proc echoLine(): void = discard + + proc no_IO_please() {.forbids: [IO].} = + # this is OK because it didn't define any tag: + echoLine() + # the compiler prevents this: + let y = readLine() + ``` + +The `forbids` pragma defines a list of illegal effects - if any statement +invokes any of those effects, the compilation will fail. +Procedure types with any disallowed effect are the subtypes of equal +procedure types without such lists: + + ```nim + type MyEffect = object + type ProcType1 = proc (i: int): void {.forbids: [MyEffect].} + type ProcType2 = proc (i: int): void + + proc caller1(p: ProcType1): void = p(1) + proc caller2(p: ProcType2): void = p(1) + + proc effectful(i: int): void {.tags: [MyEffect].} = echo $i + proc effectless(i: int): void {.forbids: [MyEffect].} = echo $i + + proc toBeCalled1(i: int): void = effectful(i) + proc toBeCalled2(i: int): void = effectless(i) + + ## this will fail because toBeCalled1 uses MyEffect which was forbidden by ProcType1: + caller1(toBeCalled1) + ## this is OK because both toBeCalled2 and ProcType1 have the same requirements: + caller1(toBeCalled2) + ## these are OK because ProcType2 doesn't have any effect requirement: + caller2(toBeCalled1) + caller2(toBeCalled2) + ``` + +`ProcType2` is a subtype of `ProcType1`. Unlike with the `tags` pragma, the parent context - the +function which calls other functions with forbidden effects - doesn't inherit the forbidden list of effects. + + +Side effects +------------ + +The `noSideEffect` pragma is used to mark a proc/iterator that can have only +side effects through parameters. This means that the proc/iterator only changes locations that are +reachable from its parameters and the return value only depends on the +parameters. If none of its parameters have the type `var`, `ref`, `ptr`, `cstring`, or `proc`, +then no locations are modified. + +In other words, a routine has no side effects if it does not access a threadlocal +or global variable and it does not call any routine that has a side effect. + +It is a static error to mark a proc/iterator to have no side effect if the compiler cannot verify this. + +As a special semantic rule, the built-in [debugEcho]( +system.html#debugEcho,varargs[typed,]) pretends to be free of side effects +so that it can be used for debugging routines marked as `noSideEffect`. + +`func` is syntactic sugar for a proc with no side effects: + + ```nim + func `+` (x, y: int): int + ``` + + +To override the compiler's side effect analysis a `{.noSideEffect.}` +`cast` pragma block can be used: + + ```nim + func f() = + {.cast(noSideEffect).}: + echo "test" + ``` + +**Side effects are usually inferred. The inference for side effects is +analogous to the inference for exception tracking.** + +When the compiler cannot infer side effects, as is the case for imported +functions, one can annotate them with the `sideEffect` pragma. + +GC safety effect +---------------- + +We call a proc `p` `GC safe`:idx: when it doesn't access any global variable +that contains GC'ed memory (`string`, `seq`, `ref` or a closure) either +directly or indirectly through a call to a GC unsafe proc. + +**The GC safety property is usually inferred. The inference for GC safety is +analogous to the inference for exception tracking.** + +The `gcsafe`:idx: annotation can be used to mark a proc to be gcsafe, +otherwise this property is inferred by the compiler. Note that `noSideEffect` +implies `gcsafe`. + +Routines that are imported from C are always assumed to be `gcsafe`. + +To override the compiler's gcsafety analysis a `{.cast(gcsafe).}` pragma block can +be used: + + ```nim + var + someGlobal: string = "some string here" + perThread {.threadvar.}: string + + proc setPerThread() = + {.cast(gcsafe).}: + deepCopy(perThread, someGlobal) + ``` + + +See also: + +- [Shared heap memory management](mm.html). + Effects pragma -------------- -The ``effects`` pragma has been designed to assist the programmer with the +The `effects` pragma has been designed to assist the programmer with the effects analysis. It is a statement that makes the compiler output all inferred -effects up to the ``effects``'s position: +effects up to the `effects`'s position: -.. code-block:: nim + ```nim proc p(what: bool) = if what: raise newException(IOError, "IO") {.effects.} else: raise newException(OSError, "OS") + ``` -The compiler produces a hint message that ``IOError`` can be raised. ``OSError`` -is not listed as it cannot be raised in the branch the ``effects`` pragma +The compiler produces a hint message that `IOError` can be raised. `OSError` +is not listed as it cannot be raised in the branch the `effects` pragma appears in. @@ -4528,17 +5524,16 @@ Generics ======== Generics are Nim's means to parametrize procs, iterators or types with -`type parameters`:idx:. Depending on context, the brackets are used either to -introduce type parameters or to instantiate a generic proc, iterator or type. +`type parameters`:idx:. Depending on the context, the brackets are used either to +introduce type parameters or to instantiate a generic proc, iterator, or type. -The following example shows a generic binary tree can be modelled: -.. code-block:: nim - :test: "nim c $1" +The following example shows how a generic binary tree can be modeled: + ```nim test = "nim c $1" type BinaryTree*[T] = ref object # BinaryTree is a generic type with - # generic param ``T`` + # generic parameter `T` le, ri: BinaryTree[T] # left and right subtrees; may be nil data: T # the data stored in a node @@ -4553,8 +5548,8 @@ The following example shows a generic binary tree can be modelled: else: var it = root while it != nil: - # compare the data items; uses the generic ``cmp`` proc - # that works for any type that has a ``==`` and ``<`` operator + # compare the data items; uses the generic `cmp` proc + # that works for any type that has a `==` and `<` operator var c = cmp(it.data, n.data) if c < 0: if it.le == nil: @@ -4573,8 +5568,8 @@ The following example shows a generic binary tree can be modelled: iterator preorder*[T](root: BinaryTree[T]): T = # Preorder traversal of a binary tree. - # Since recursive iterators are not yet implemented, - # this uses an explicit stack (which is more efficient anyway): + # This uses an explicit stack (which is more efficient than + # a recursive iterator factory). var stack: seq[BinaryTree[T]] = @[root] while stack.len > 0: var n = stack.pop() @@ -4584,56 +5579,99 @@ The following example shows a generic binary tree can be modelled: n = n.le # and follow the left pointer var - root: BinaryTree[string] # instantiate a BinaryTree with ``string`` - add(root, newNode("hello")) # instantiates ``newNode`` and ``add`` - add(root, "world") # instantiates the second ``add`` proc + root: BinaryTree[string] # instantiate a BinaryTree with `string` + add(root, newNode("hello")) # instantiates `newNode` and `add` + add(root, "world") # instantiates the second `add` proc for str in preorder(root): stdout.writeLine(str) + ``` -The ``T`` is called a `generic type parameter`:idx: or +The `T` is called a `generic type parameter`:idx: or a `type variable`:idx:. +Generic Procs +--------------- + +Let's consider the anatomy of a generic `proc` to agree on defined terminology. + +```nim +p[T: t](arg1: f): y +``` + +- `p`: Callee symbol +- `[...]`: Generic parameters +- `T: t`: Generic constraint +- `T`: Type variable +- `[T: t](arg1: f): y`: Formal signature +- `arg1: f`: Formal parameter +- `f`: Formal parameter type +- `y`: Formal return type + +The use of the word "formal" here is to denote the symbols as they are defined by the programmer, +not as they may be at compile time contextually. Since generics may be instantiated and +types bound, we have more than one entity to think about when generics are involved. + +The usage of a generic will resolve the formally defined expression into an instance of that +expression bound to only concrete types. This process is called "instantiation". + +Brackets at the site of a generic's formal definition specify the "constraints" as in: + +```nim +type Foo[T] = object +proc p[H;T: Foo[H]](param: T): H +``` + +A constraint definition may have more than one symbol defined by separating each definition by +a `;`. Notice how `T` is composed of `H` and the return type of `p` is defined as `H`. When this +generic proc is instantiated `H` will be bound to a concrete type, thus making `T` concrete and +the return type of `p` will be bound to the same concrete type used to define `H`. + +Brackets at the site of usage can be used to supply concrete types to instantiate the generic in the same +order that the symbols are defined in the constraint. Alternatively, type bindings may be inferred by the compiler +in some situations, allowing for cleaner code. + + Is operator ----------- -The ``is`` operator is evaluated during semantic analysis to check for type +The `is` operator is evaluated during semantic analysis to check for type equivalence. It is therefore very useful for type specialization within generic code: -.. code-block:: nim + ```nim type Table[Key, Value] = object keys: seq[Key] values: seq[Value] when not (Key is string): # empty value for strings used for optimization deletedKeys: seq[bool] + ``` -Type Classes +Type classes ------------ A type class is a special pseudo-type that can be used to match against -types in the context of overload resolution or the ``is`` operator. +types in the context of overload resolution or the `is` operator. Nim supports the following built-in type classes: ================== =================================================== type class matches ================== =================================================== -``object`` any object type -``tuple`` any tuple type - -``enum`` any enumeration -``proc`` any proc type -``ref`` any ``ref`` type -``ptr`` any ``ptr`` type -``var`` any ``var`` type -``distinct`` any distinct type -``array`` any array type -``set`` any set type -``seq`` any seq type -``auto`` any type -``any`` distinct auto (see below) +`object` any object type +`tuple` any tuple type +`enum` any enumeration +`proc` any proc type +`iterator` any iterator type +`ref` any `ref` type +`ptr` any `ptr` type +`var` any `var` type +`distinct` any distinct type +`array` any array type +`set` any set type +`seq` any seq type +`auto` any type ================== =================================================== Furthermore, every generic type automatically creates a type class of the same @@ -4642,38 +5680,59 @@ name that will match any instantiation of the generic type. Type classes can be combined using the standard boolean operators to form more complex type classes: -.. code-block:: nim + ```nim # create a type class that will match all tuple and object types - type RecordType = tuple or object + type RecordType = (tuple or object) proc printFields[T: RecordType](rec: T) = for key, value in fieldPairs(rec): echo key, " = ", value + ``` + +Type constraints on generic parameters can be grouped with `,` and propagation +stops with `;`, similarly to parameters for macros and templates: + ```nim + proc fn1[T; U, V: SomeFloat]() = discard # T is unconstrained + template fn2(t; u, v: SomeFloat) = discard # t is unconstrained + ``` Whilst the syntax of type classes appears to resemble that of ADTs/algebraic data types in ML-like languages, it should be understood that type classes are static constraints to be enforced at type instantiations. Type classes are not really -types in themselves, but are instead a system of providing generic "checks" that +types in themselves but are instead a system of providing generic "checks" that ultimately *resolve* to some singular type. Type classes do not allow for runtime type dynamism, unlike object variants or methods. As an example, the following would not compile: -.. code-block:: nim + ```nim type TypeClass = int | string var foo: TypeClass = 2 # foo's type is resolved to an int here foo = "this will fail" # error here, because foo is an int + ``` Nim allows for type classes and regular types to be specified as `type constraints`:idx: of the generic type parameter: -.. code-block:: nim + ```nim proc onlyIntOrString[T: int|string](x, y: T) = discard onlyIntOrString(450, 616) # valid onlyIntOrString(5.0, 0.0) # type mismatch onlyIntOrString("xy", 50) # invalid as 'T' cannot be both at the same time + ``` + +`proc` and `iterator` type classes also accept a calling convention pragma +to restrict the calling convention of the matching `proc` or `iterator` type. + + ```nim + proc onlyClosure[T: proc {.closure.}](x: T) = discard + + onlyClosure(proc() = echo "hello") # valid + proc foo() {.nimcall.} = discard + onlyClosure(foo) # type mismatch + ``` Implicit generics @@ -4681,107 +5740,108 @@ Implicit generics A type class can be used directly as the parameter's type. -.. code-block:: nim - + ```nim # create a type class that will match all tuple and object types - type RecordType = tuple or object + type RecordType = (tuple or object) proc printFields(rec: RecordType) = for key, value in fieldPairs(rec): echo key, " = ", value + ``` -Procedures utilizing type classes in such manner are considered to be +Procedures utilizing type classes in such a manner are considered to be `implicitly generic`:idx:. They will be instantiated once for each unique -combination of param types used within the program. +combination of parameter types used within the program. -By default, during overload resolution each named type class will bind to +By default, during overload resolution, each named type class will bind to exactly one concrete type. We call such type classes `bind once`:idx: types. Here is an example taken directly from the system module to illustrate this: -.. code-block:: nim + ```nim proc `==`*(x, y: tuple): bool = ## requires `x` and `y` to be of the same tuple type - ## generic ``==`` operator for tuples that is lifted from the components + ## generic `==` operator for tuples that is lifted from the components ## of `x` and `y`. result = true for a, b in fields(x, y): if a != b: result = false + ``` -Alternatively, the ``distinct`` type modifier can be applied to the type class -to allow each param matching the type class to bind to a different type. Such +Alternatively, the `distinct` type modifier can be applied to the type class +to allow each parameter matching the type class to bind to a different type. Such type classes are called `bind many`:idx: types. Procs written with the implicitly generic style will often need to refer to the type parameters of the matched generic type. They can be easily accessed using the dot syntax: -.. code-block:: nim + ```nim type Matrix[T, Rows, Columns] = object ... proc `[]`(m: Matrix, row, col: int): Matrix.T = m.data[col * high(Matrix.Columns) + row] + ``` Here are more examples that illustrate implicit generics: -.. code-block:: nim - + ```nim proc p(t: Table; k: Table.Key): Table.Value # is roughly the same as: proc p[Key, Value](t: Table[Key, Value]; k: Key): Value + ``` -.. code-block:: nim - + ```nim proc p(a: Table, b: Table) # is roughly the same as: proc p[Key, Value](a, b: Table[Key, Value]) + ``` -.. code-block:: nim - + ```nim proc p(a: Table, b: distinct Table) # is roughly the same as: proc p[Key, Value, KeyB, ValueB](a: Table[Key, Value], b: Table[KeyB, ValueB]) + ``` `typedesc` used as a parameter type also introduces an implicit generic. `typedesc` has its own set of rules: -.. code-block:: nim - + ```nim proc p(a: typedesc) # is roughly the same as: proc p[T](a: typedesc[T]) + ``` `typedesc` is a "bind many" type class: -.. code-block:: nim - + ```nim proc p(a, b: typedesc) # is roughly the same as: proc p[T, T2](a: typedesc[T], b: typedesc[T2]) + ``` A parameter of type `typedesc` is itself usable as a type. If it is used -as a type, it's the underlying type. (In other words, one level +as a type, it's the underlying type. In other words, one level of "typedesc"-ness is stripped off: -.. code-block:: nim - + ```nim proc p(a: typedesc; b: a) = discard # is roughly the same as: @@ -4790,18 +5850,16 @@ of "typedesc"-ness is stripped off: # hence this is a valid call: p(int, 4) # as parameter 'a' requires a type, but 'b' requires a value. + ``` Generic inference restrictions ------------------------------ -The types ``var T``, ``out T`` and ``typedesc[T]`` cannot be inferred in a generic +The types `var T` and `typedesc[T]` cannot be inferred in a generic instantiation. The following is not allowed: -.. code-block:: nim - :test: "nim c $1" - :status: 1 - + ```nim test = "nim c $1" status = 1 proc g[T](f: proc(x: T); x: T) = f(x) @@ -4818,26 +5876,24 @@ instantiation. The following is not allowed: # also not allowed: explicit instantiation via 'var int' g[var int](v, i) + ``` Symbol lookup in generics ------------------------- -Open and Closed symbols -~~~~~~~~~~~~~~~~~~~~~~~ +### Open and Closed symbols The symbol binding rules in generics are slightly subtle: There are "open" and "closed" symbols. A "closed" symbol cannot be re-bound in the instantiation -context, an "open" symbol can. Per default overloaded symbols are open +context, an "open" symbol can. Per default, overloaded symbols are open and every other symbol is closed. Open symbols are looked up in two different contexts: Both the context at definition and the context at instantiation are considered: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" type Index = distinct int @@ -4847,10 +5903,11 @@ at definition and the context at instantiation are considered: var b = (0, 0.Index) echo a == b # works! + ``` -In the example the generic ``==`` for tuples (as defined in the system module) -uses the ``==`` operators of the tuple's components. However, the ``==`` for -the ``Index`` type is defined *after* the ``==`` for tuples; yet the example +In the example, the [generic `==` for tuples](system.html#%3D%3D%2CT%2CT_2) (as defined in the system module) +uses the `==` operators of the tuple's components. However, the `==` for +the `Index` type is defined *after* the `==` for tuples; yet the example compiles as the instantiation takes the currently defined symbols into account too. @@ -4859,28 +5916,27 @@ Mixin statement A symbol can be forced to be open by a `mixin`:idx: declaration: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" proc create*[T](): ref T = # there is no overloaded 'init' here, so we need to state that it's an # open symbol explicitly: mixin init new result init result + ``` -``mixin`` statements only make sense in templates and generics. +`mixin` statements only make sense in templates and generics. Bind statement -------------- -The ``bind`` statement is the counterpart to the ``mixin`` statement. It +The `bind` statement is the counterpart to the `mixin` statement. It can be used to explicitly declare identifiers that should be bound early (i.e. the identifiers should be looked up in the scope of the template/generic definition): -.. code-block:: nim + ```nim # Module A var lastId = 0 @@ -4889,17 +5945,63 @@ definition): bind lastId inc(lastId) lastId + ``` -.. code-block:: nim + ```nim # Module B import A echo genId() + ``` -But a ``bind`` is rarely useful because symbol binding from the definition +But a `bind` is rarely useful because symbol binding from the definition scope is the default. -``bind`` statements only make sense in templates and generics. +`bind` statements only make sense in templates and generics. + + +Delegating bind statements +-------------------------- + +The following example outlines a problem that can arise when generic +instantiations cross multiple different modules: + + ```nim + # module A + proc genericA*[T](x: T) = + mixin init + init(x) + ``` + + + ```nim + import C + + # module B + proc genericB*[T](x: T) = + # Without the `bind init` statement C's init proc is + # not available when `genericB` is instantiated: + bind init + genericA(x) + ``` + + ```nim + # module C + type O = object + proc init*(x: var O) = discard + ``` + + ```nim + # module main + import B, C + + genericB O() + ``` + +In module B has an `init` proc from module C in its scope that is not +taken into account when `genericB` is instantiated which leads to the +instantiation of `genericA`. The solution is to `forward`:idx: these +symbols by a `bind` statement inside `genericB`. Templates @@ -4913,71 +6015,67 @@ The syntax to *invoke* a template is the same as calling a procedure. Example: -.. code-block:: nim + ```nim template `!=` (a, b: untyped): untyped = - # this definition exists in the System module + # this definition exists in the system module not (a == b) assert(5 != 6) # the compiler rewrites that to: assert(not (5 == 6)) + ``` -The ``!=``, ``>``, ``>=``, ``in``, ``notin``, ``isnot`` operators are in fact +The `!=`, `>`, `>=`, `in`, `notin`, `isnot` operators are in fact templates: -| ``a > b`` is transformed into ``b < a``. -| ``a in b`` is transformed into ``contains(b, a)``. -| ``notin`` and ``isnot`` have the obvious meanings. +| `a > b` is transformed into `b < a`. +| `a in b` is transformed into `contains(b, a)`. +| `notin` and `isnot` have the obvious meanings. -The "types" of templates can be the symbols ``untyped``, -``typed`` or ``typedesc``. These are "meta types", they can only be used in certain -contexts. Regular types can be used too; this implies that ``typed`` expressions +The "types" of templates can be the symbols `untyped`, +`typed` or `typedesc`. These are "meta types", they can only be used in certain +contexts. Regular types can be used too; this implies that `typed` expressions are expected. Typed vs untyped parameters --------------------------- -An ``untyped`` parameter means that symbol lookups and type resolution is not -performed before the expression is passed to the template. This means that for -example *undeclared* identifiers can be passed to the template: - -.. code-block:: nim - :test: "nim c $1" +An `untyped` parameter means that symbol lookups and type resolution is not +performed before the expression is passed to the template. This means that +*undeclared* identifiers, for example, can be passed to the template: + ```nim test = "nim c $1" template declareInt(x: untyped) = var x: int declareInt(x) # valid x = 3 + ``` -.. code-block:: nim - :test: "nim c $1" - :status: 1 - + ```nim test = "nim c $1" status = 1 template declareInt(x: typed) = var x: int - declareInt(x) # invalid, because x has not been declared and so has no type + declareInt(x) # invalid, because x has not been declared and so it has no type + ``` -A template where every parameter is ``untyped`` is called an `immediate`:idx: -template. For historical reasons templates can be explicitly annotated with -an ``immediate`` pragma and then these templates do not take part in +A template where every parameter is `untyped` is called an `immediate`:idx: +template. For historical reasons, templates can be explicitly annotated with +an `immediate` pragma and then these templates do not take part in overloading resolution and the parameters' types are *ignored* by the compiler. Explicit immediate templates are now deprecated. -**Note**: For historical reasons ``stmt`` was an alias for ``typed`` and -``expr`` was an alias for ``untyped``, but they are removed. +**Note**: For historical reasons, `stmt` was an alias for `typed` and +`expr` was an alias for `untyped`, but they are removed. Passing a code block to a template ---------------------------------- One can pass a block of statements as the last argument to a template -following the special ``:`` syntax: - -.. code-block:: nim - :test: "nim c $1" +following the special `:` syntax: + ```nim test = "nim c $1" template withFile(f, fn, mode, actions: untyped): untyped = var f: File if open(f, fn, mode): @@ -4991,68 +6089,54 @@ following the special ``:`` syntax: withFile(txt, "ttempl3.txt", fmWrite): # special colon txt.writeLine("line 1") txt.writeLine("line 2") + ``` -In the example, the two ``writeLine`` statements are bound to the ``actions`` +In the example, the two `writeLine` statements are bound to the `actions` parameter. -Usually to pass a block of code to a template the parameter that accepts -the block needs to be of type ``untyped``. Because symbol lookups are then +Usually, to pass a block of code to a template, the parameter that accepts +the block needs to be of type `untyped`. Because symbol lookups are then delayed until template instantiation time: -.. code-block:: nim - :test: "nim c $1" - :status: 1 - + ```nim test = "nim c $1" status = 1 template t(body: typed) = + proc p = echo "hey" block: body t: - var i = 1 - echo i + p() # fails with 'undeclared identifier: p' + ``` - t: - var i = 2 # fails with 'attempt to redeclare i' - echo i - -The above code fails with the mysterious error message that ``i`` has already -been declared. The reason for this is that the ``var i = ...`` bodies need to -be type-checked before they are passed to the ``body`` parameter and type -checking in Nim implies symbol lookups. For the symbol lookups to succeed -``i`` needs to be added to the current (i.e. outer) scope. After type checking -these additions to the symbol table are not rolled back (for better or worse). -The same code works with ``untyped`` as the passed body is not required to be +The above code fails with the error message that `p` is not declared. +The reason for this is that the `p()` body is type-checked before getting +passed to the `body` parameter and type checking in Nim implies symbol lookups. +The same code works with `untyped` as the passed body is not required to be type-checked: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" template t(body: untyped) = + proc p = echo "hey" block: body t: - var i = 1 - echo i - - t: - var i = 2 # compiles - echo i + p() # compiles + ``` Varargs of untyped ------------------ -In addition to the ``untyped`` meta-type that prevents type checking there is -also ``varargs[untyped]`` so that not even the number of parameters is fixed: - -.. code-block:: nim - :test: "nim c $1" +In addition to the `untyped` meta-type that prevents type checking, there is +also `varargs[untyped]` so that not even the number of parameters is fixed: + ```nim test = "nim c $1" template hideIdentifiers(x: varargs[untyped]) = discard hideIdentifiers(undeclared1, undeclared2) + ``` However, since a template cannot iterate over varargs, this feature is generally much more useful for macros. @@ -5064,7 +6148,7 @@ Symbol binding in templates A template is a `hygienic`:idx: macro and so opens a new scope. Most symbols are bound from the definition scope of the template: -.. code-block:: nim + ```nim # Module A var lastId = 0 @@ -5072,14 +6156,16 @@ bound from the definition scope of the template: template genId*: untyped = inc(lastId) lastId + ``` -.. code-block:: nim + ```nim # Module B import A echo genId() # Works as 'lastId' has been bound in 'genId's defining scope + ``` -As in generics symbol binding can be influenced via ``mixin`` or ``bind`` +As in generics, symbol binding can be influenced via `mixin` or `bind` statements. @@ -5087,11 +6173,9 @@ statements. Identifier construction ----------------------- -In templates identifiers can be constructed with the backticks notation: - -.. code-block:: nim - :test: "nim c $1" +In templates, identifiers can be constructed with the backticks notation: + ```nim test = "nim c $1" template typedef(name: untyped, typ: typedesc) = type `T name`* {.inject.} = typ @@ -5099,19 +6183,20 @@ In templates identifiers can be constructed with the backticks notation: typedef(myint, int) var x: PMyInt + ``` -In the example ``name`` is instantiated with ``myint``, so \`T name\` becomes -``Tmyint``. +In the example, `name` is instantiated with `myint`, so \`T name\` becomes +`Tmyint`. Lookup rules for template parameters ------------------------------------ -A parameter ``p`` in a template is even substituted in the expression ``x.p``. -Thus template arguments can be used as field names and a global symbol can be +A parameter `p` in a template is even substituted in the expression `x.p`. +Thus, template arguments can be used as field names and a global symbol can be shadowed by the same argument name even when fully qualified: -.. code-block:: nim + ```nim # module 'm' type @@ -5125,10 +6210,11 @@ shadowed by the same argument name even when fully qualified: tstLev(levA) # produces: 'levA levA' + ``` -But the global symbol can properly be captured by a ``bind`` statement: +But the global symbol can properly be captured by a `bind` statement: -.. code-block:: nim + ```nim # module 'm' type @@ -5143,17 +6229,16 @@ But the global symbol can properly be captured by a ``bind`` statement: tstLev(levA) # produces: 'levA levB' + ``` Hygiene in templates -------------------- -Per default templates are `hygienic`:idx:\: Local identifiers declared in a +Per default, templates are `hygienic`:idx:\: Local identifiers declared in a template cannot be accessed in the instantiation context: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" template newException*(exceptn: typedesc, message: string): untyped = var e: ref exceptn # e is implicitly gensym'ed here @@ -5164,50 +6249,54 @@ template cannot be accessed in the instantiation context: # so this works: let e = "message" raise newException(IoError, e) + ``` Whether a symbol that is declared in a template is exposed to the instantiation -scope is controlled by the `inject`:idx: and `gensym`:idx: pragmas: gensym'ed -symbols are not exposed but inject'ed are. +scope is controlled by the `inject`:idx: and `gensym`:idx: pragmas: +`gensym`'ed symbols are not exposed but `inject`'ed symbols are. -The default for symbols of entity ``type``, ``var``, ``let`` and ``const`` -is ``gensym`` and for ``proc``, ``iterator``, ``converter``, ``template``, -``macro`` is ``inject``. However, if the name of the entity is passed as a -template parameter, it is an inject'ed symbol: +The default for symbols of entity `type`, `var`, `let` and `const` +is `gensym`. For `proc`, `iterator`, `converter`, `template`, +`macro`, the default is `inject`, but if a `gensym` symbol with the same name +is defined in the same syntax-level scope, it will be `gensym` by default. +This can be overriden by marking the routine as `inject`. -.. code-block:: nim +If the name of the entity is passed as a template parameter, it is an `inject`'ed symbol: + + ```nim template withFile(f, fn, mode: untyped, actions: untyped): untyped = block: - var f: File # since 'f' is a template param, it's injected implicitly + var f: File # since 'f' is a template parameter, it's injected implicitly ... withFile(txt, "ttempl3.txt", fmWrite): txt.writeLine("line 1") txt.writeLine("line 2") + ``` -The ``inject`` and ``gensym`` pragmas are second class annotations; they have -no semantics outside of a template definition and cannot be abstracted over: +The `inject` and `gensym` pragmas are second class annotations; they have +no semantics outside a template definition and cannot be abstracted over: -.. code-block:: nim + ```nim {.pragma myInject: inject.} template t() = var x {.myInject.}: int # does NOT work + ``` To get rid of hygiene in templates, one can use the `dirty`:idx: pragma for -a template. ``inject`` and ``gensym`` have no effect in ``dirty`` templates. +a template. `inject` and `gensym` have no effect in `dirty` templates. -``gensym``'ed symbols cannot be used as ``field`` in the ``x.field`` syntax. -Nor can they be used in the ``ObjectConstruction(field: value)`` -and ``namedParameterCall(field = value)`` syntactic constructs. +`gensym`'ed symbols cannot be used as `field` in the `x.field` syntax. +Nor can they be used in the `ObjectConstruction(field: value)` +and `namedParameterCall(field = value)` syntactic constructs. The reason for this is that code like -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" type T = object f: int @@ -5215,17 +6304,15 @@ The reason for this is that code like template tmp(x: T) = let f = 34 echo x.f, T(f: 4) + ``` should work as expected. However, this means that the method call syntax is not available for -``gensym``'ed symbols: - -.. code-block:: nim - :test: "nim c $1" - :status: 1 +`gensym`'ed symbols: + ```nim test = "nim c $1" status = 1 template tmp(x) = type T {.gensym.} = int @@ -5233,65 +6320,39 @@ However, this means that the method call syntax is not available for echo x.T # invalid: instead use: 'echo T(x)'. tmp(12) - - -**Note**: The Nim compiler prior to version 1 was more lenient about this -requirement. Use the ``--useVersion:0.19`` switch for a transition period. - + ``` Limitations of the method call syntax ------------------------------------- -The expression ``x`` in ``x.f`` needs to be semantically checked (that means +The expression `x` in `x.f` needs to be semantically checked (that means symbol lookup and type checking) before it can be decided that it needs to be -rewritten to ``f(x)``. Therefore the dot syntax has some limitations when it +rewritten to `f(x)`. Therefore, the dot syntax has some limitations when it is used to invoke templates/macros: -.. code-block:: nim - :test: "nim c $1" - :status: 1 - + ```nim test = "nim c $1" status = 1 template declareVar(name: untyped) = const name {.inject.} = 45 # Doesn't compile: unknownIdentifier.declareVar + ``` -Another common example is this: - -.. code-block:: nim - :test: "nim c $1" - :status: 1 - - from sequtils import toSeq - - iterator something: string = - yield "Hello" - yield "World" - - var info = something().toSeq - -The problem here is that the compiler already decided that ``something()`` as -an iterator is not callable in this context before ``toSeq`` gets its -chance to convert it into a sequence. - It is also not possible to use fully qualified identifiers with module symbol in method call syntax. The order in which the dot operator binds to symbols prohibits this. -.. code-block:: nim - :test: "nim c $1" - :status: 1 + ```nim test = "nim c $1" status = 1 + import std/sequtils - import sequtils - - var myItems = @[1,3,3,7] - let N1 = count(myItems, 3) # OK - let N2 = sequtils.count(myItems, 3) # fully qualified, OK - let N3 = myItems.count(3) # OK - let N4 = myItems.sequtils.count(3) # illegal, `myItems.sequtils` can't be resolved + var myItems = @[1,3,3,7] + let N1 = count(myItems, 3) # OK + let N2 = sequtils.count(myItems, 3) # fully qualified, OK + let N3 = myItems.count(3) # OK + let N4 = myItems.sequtils.count(3) # illegal, `myItems.sequtils` can't be resolved + ``` This means that when for some reason a procedure needs a disambiguation through the module name, the call needs to be @@ -5301,10 +6362,10 @@ Macros ====== A macro is a special function that is executed at compile time. -Normally the input for a macro is an abstract syntax +Normally, the input for a macro is an abstract syntax tree (AST) of the code that is passed to it. The macro can then do transformations on it and return the transformed AST. This can be used to -add custom language features and implement `domain specific languages`:idx:. +add custom language features and implement `domain-specific languages`:idx:. Macro invocation is a case where semantic analysis does **not** entirely proceed top to bottom and left to right. Instead, semantic analysis happens at least @@ -5320,18 +6381,24 @@ twice: While macros enable advanced compile-time code transformations, they cannot change Nim's syntax. -Debug Example +**Style note:** For code readability, it is best to use the least powerful +programming construct that remains expressive. So the "check list" is: + +(1) Use an ordinary proc/iterator, if possible. +(2) Else: Use a generic proc/iterator, if possible. +(3) Else: Use a template, if possible. +(4) Else: Use a macro. + +Debug example ------------- -The following example implements a powerful ``debug`` command that accepts a +The following example implements a powerful `debug` command that accepts a variable number of arguments: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" # to work with Nim syntax trees, we need an API that is defined in the - # ``macros`` module: - import macros + # `macros` module: + import std/macros macro debug(args: varargs[untyped]): untyped = # `args` is a collection of `NimNode` values that each contain the @@ -5356,10 +6423,11 @@ variable number of arguments: a[1] = 45 debug(a[0], a[1], x) + ``` The macro call expands to: -.. code-block:: nim + ```nim write(stdout, "a[0]") write(stdout, ": ") writeLine(stdout, a[0]) @@ -5371,26 +6439,25 @@ The macro call expands to: write(stdout, "x") write(stdout, ": ") writeLine(stdout, x) + ``` -Arguments that are passed to a ``varargs`` parameter are wrapped in an array -constructor expression. This is why ``debug`` iterates over all of ``n``'s +Arguments that are passed to a `varargs` parameter are wrapped in an array +constructor expression. This is why `debug` iterates over all of `args`'s children. -BindSym +bindSym ------- -The above ``debug`` macro relies on the fact that ``write``, ``writeLine`` and -``stdout`` are declared in the system module and thus visible in the +The above `debug` macro relies on the fact that `write`, `writeLine` and +`stdout` are declared in the system module and are thus visible in the instantiating context. There is a way to use bound identifiers -(aka `symbols`:idx:) instead of using unbound identifiers. The ``bindSym`` +(aka `symbols`:idx:) instead of using unbound identifiers. The `bindSym` builtin can be used for that: -.. code-block:: nim - :test: "nim c $1" - - import macros + ```nim test = "nim c $1" + import std/macros macro debug(n: varargs[typed]): untyped = result = newNimNode(nnkStmtList, n) @@ -5407,10 +6474,11 @@ builtin can be used for that: a[1] = 45 debug(a[0], a[1], x) + ``` The macro call expands to: -.. code-block:: nim + ```nim write(stdout, "a[0]") write(stdout, ": ") writeLine(stdout, a[0]) @@ -5422,60 +6490,160 @@ The macro call expands to: write(stdout, "x") write(stdout, ": ") writeLine(stdout, x) + ``` -However, the symbols ``write``, ``writeLine`` and ``stdout`` are already bound -and are not looked up again. As the example shows, ``bindSym`` does work with -overloaded symbols implicitly. - -Case-Of Macro -------------- - -In Nim it is possible to have a macro with the syntax of a *case-of* -expression just with the difference that all of branches are passed to -and processed by the macro implementation. It is then up the macro -implementation to transform the *of-branches* into a valid Nim -statement. The following example should show how this feature could be -used for a lexical analyzer. +In this version of `debug`, the symbols `write`, `writeLine` and `stdout` +are already bound and are not looked up again. As the example shows, `bindSym` +does work with overloaded symbols implicitly. -.. code-block:: nim - import macros +Note that the symbol names passed to `bindSym` have to be constant. The +experimental feature `dynamicBindSym` ([experimental manual]( +manual_experimental.html#dynamic-arguments-for-bindsym)) +allows this value to be computed dynamically. - macro case_token(args: varargs[untyped]): untyped = - echo args.treeRepr - # creates a lexical analyzer from regular expressions - # ... (implementation is an exercise for the reader ;-) - discard +Post-statement blocks +--------------------- - case_token: # this colon tells the parser it is a macro statement - of r"[A-Za-z_]+[A-Za-z_0-9]*": - return tkIdentifier - of r"0-9+": - return tkInteger - of r"[\+\-\*\?]+": - return tkOperator +Macros can receive `of`, `elif`, `else`, `except`, `finally` and `do` +blocks (including their different forms such as `do` with routine parameters) +as arguments if called in statement form. + + ```nim + macro performWithUndo(task, undo: untyped) = ... + + performWithUndo do: + # multiple-line block of code + # to perform the task + do: + # code to undo it + + let num = 12 + # a single colon may be used if there is no initial block + match (num mod 3, num mod 5): + of (0, 0): + echo "FizzBuzz" + of (0, _): + echo "Fizz" + of (_, 0): + echo "Buzz" else: - return tkUnknown + echo num + ``` -**Style note**: For code readability, it is the best idea to use the least -powerful programming construct that still suffices. So the "check list" is: +For loop macro +-------------- -(1) Use an ordinary proc/iterator, if possible. -(2) Else: Use a generic proc/iterator, if possible. -(3) Else: Use a template, if possible. -(4) Else: Use a macro. +A macro that takes as its only input parameter an expression of the special +type `system.ForLoopStmt` can rewrite the entirety of a `for` loop: + + ```nim test = "nim c $1" + import std/macros + + macro example(loop: ForLoopStmt) = + result = newTree(nnkForStmt) # Create a new For loop. + result.add loop[^3] # This is "item". + result.add loop[^2][^1] # This is "[1, 2, 3]". + result.add newCall(bindSym"echo", loop[0]) + + for item in example([1, 2, 3]): discard + ``` + +Expands to: + + ```nim + for item in items([1, 2, 3]): + echo item + ``` + +Another example: + + ```nim test = "nim c $1" + import std/macros + + macro enumerate(x: ForLoopStmt): untyped = + expectKind x, nnkForStmt + # check if the starting count is specified: + var countStart = if x[^2].len == 2: newLit(0) else: x[^2][1] + result = newStmtList() + # we strip off the first for loop variable and use it as an integer counter: + result.add newVarStmt(x[0], countStart) + var body = x[^1] + if body.kind != nnkStmtList: + body = newTree(nnkStmtList, body) + body.add newCall(bindSym"inc", x[0]) + var newFor = newTree(nnkForStmt) + for i in 1..x.len-3: + newFor.add x[i] + # transform enumerate(X) to 'X' + newFor.add x[^2][^1] + newFor.add body + result.add newFor + # now wrap the whole macro in a block to create a new scope + result = quote do: + block: `result` + + for a, b in enumerate(items([1, 2, 3])): + echo a, " ", b + + # without wrapping the macro in a block, we'd need to choose different + # names for `a` and `b` here to avoid redefinition errors + for a, b in enumerate(10, [1, 2, 3, 5]): + echo a, " ", b + ``` + + +Case statement macros +--------------------- + +Macros named `` `case` `` can provide implementations of `case` statements +for certain types. The following is an example of such an implementation +for tuples, leveraging the existing equality operator for tuples +(as provided in `system.==`): + + ```nim test = "nim c $1" + import std/macros + + macro `case`(n: tuple): untyped = + result = newTree(nnkIfStmt) + let selector = n[0] + for i in 1 ..< n.len: + let it = n[i] + case it.kind + of nnkElse, nnkElifBranch, nnkElifExpr, nnkElseExpr: + result.add it + of nnkOfBranch: + for j in 0..it.len-2: + let cond = newCall("==", selector, it[j]) + result.add newTree(nnkElifBranch, cond, it[^1]) + else: + error "custom 'case' for tuple cannot handle this node", it + + case ("foo", 78) + of ("foo", 78): echo "yes" + of ("bar", 88): echo "no" + else: discard + ``` + +`case` macros are subject to overload resolution. The type of the +`case` statement's selector expression is matched against the type +of the first argument of the `case` macro. Then the complete `case` +statement is passed in place of the argument and the macro is evaluated. + +In other words, the macro needs to transform the full `case` statement +but only the statement's selector expression is used to determine which +macro to call. Special Types ============= -static[T] ---------- +static\[T] +---------- As their name suggests, static parameters must be constant expressions: -.. code-block:: nim - + ```nim proc precompiledRegex(pattern: static string): RegEx = var res {.global.} = re(pattern) return res @@ -5485,16 +6653,16 @@ As their name suggests, static parameters must be constant expressions: precompiledRegex(paramStr(1)) # Error, command-line options # are not constant expressions + ``` -For the purposes of code generation, all static params are treated as -generic params - the proc will be compiled separately for each unique +For the purposes of code generation, all static parameters are treated as +generic parameters - the proc will be compiled separately for each unique supplied value (or combination of values). -Static params can also appear in the signatures of generic types: - -.. code-block:: nim +Static parameters can also appear in the signatures of generic types: + ```nim type Matrix[M,N: static int; T: Number] = array[0..(M*N - 1), T] # Note how `Number` is just a type constraint here, while @@ -5505,74 +6673,74 @@ Static params can also appear in the signatures of generic types: var m1: AffineTransform3D[float] # OK var m2: AffineTransform2D[string] # Error, `string` is not a `Number` + ``` -Please note that ``static T`` is just a syntactic convenience for the underlying -generic type ``static[T]``. The type param can be omitted to obtain the type +Please note that `static T` is just a syntactic convenience for the underlying +generic type `static[T]`. The type parameter can be omitted to obtain the type class of all constant expressions. A more specific type class can be created by -instantiating ``static`` with another type class. +instantiating `static` with another type class. One can force an expression to be evaluated at compile time as a constant -expression by coercing it to a corresponding ``static`` type: +expression by coercing it to a corresponding `static` type: -.. code-block:: nim - import math + ```nim + import std/math echo static(fac(5)), " ", static[bool](16.isPowerOfTwo) + ``` The compiler will report any failure to evaluate the expression or a possible type mismatch error. -typedesc[T] ------------ +typedesc\[T] +------------ -In many contexts, Nim allows to treat the names of types as regular -values. These values exists only during the compilation phase, but since -all values must have a type, ``typedesc`` is considered their special type. +In many contexts, Nim treats the names of types as regular +values. These values exist only during the compilation phase, but since +all values must have a type, `typedesc` is considered their special type. -``typedesc`` acts like a generic type. For instance, the type of the symbol -``int`` is ``typedesc[int]``. Just like with regular generic types, when the -generic param is omitted, ``typedesc`` denotes the type class of all types. -As a syntactic convenience, one can also use ``typedesc`` as a modifier. +`typedesc` acts as a generic type. For instance, the type of the symbol +`int` is `typedesc[int]`. Just like with regular generic types, when the +generic parameter is omitted, `typedesc` denotes the type class of all types. +As a syntactic convenience, one can also use `typedesc` as a modifier. -Procs featuring ``typedesc`` params are considered implicitly generic. -They will be instantiated for each unique combination of supplied types -and within the body of the proc, the name of each param will refer to +Procs featuring `typedesc` parameters are considered implicitly generic. +They will be instantiated for each unique combination of supplied types, +and within the body of the proc, the name of each parameter will refer to the bound concrete type: -.. code-block:: nim - + ```nim proc new(T: typedesc): ref T = echo "allocating ", T.name new(result) var n = Node.new var tree = new(BinaryTree[int]) + ``` -When multiple type params are present, they will bind freely to different -types. To force a bind-once behavior one can use an explicit generic param: +When multiple type parameters are present, they will bind freely to different +types. To force a bind-once behavior, one can use an explicit generic parameter: -.. code-block:: nim + ```nim proc acceptOnlyTypePairs[T, U](A, B: typedesc[T]; C, D: typedesc[U]) + ``` -Once bound, type params can appear in the rest of the proc signature: - -.. code-block:: nim - :test: "nim c $1" +Once bound, type parameters can appear in the rest of the proc signature: + ```nim test = "nim c $1" template declareVariableWithType(T: typedesc, value: T) = var x: T = value declareVariableWithType int, 42 + ``` Overload resolution can be further influenced by constraining the set -of types that will match the type param. This works in practice to +of types that will match the type parameter. This works in practice by attaching attributes to types via templates. The constraint can be a concrete type or a type class. -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" template maxval(T: typedesc[int]): int = high(int) template maxval(T: typedesc[float]): float = Inf @@ -5588,54 +6756,54 @@ concrete type or a type class. echo "is int a number? ", isNumber(int) echo "is float a number? ", isNumber(float) echo "is RootObj a number? ", isNumber(RootObj) + ``` -Passing ``typedesc`` almost identical, just with the differences that +Passing `typedesc` is almost identical, just with the difference that the macro is not instantiated generically. The type expression is -simply passed as a ``NimNode`` to the macro, like everything else. - -.. code-block:: nim +simply passed as a `NimNode` to the macro, like everything else. - import macros + ```nim + import std/macros macro forwardType(arg: typedesc): typedesc = - # ``arg`` is of type ``NimNode`` + # `arg` is of type `NimNode` let tmp: NimNode = arg result = tmp var tmp: forwardType(int) + ``` typeof operator --------------- -**Note**: ``typeof(x)`` can for historical reasons also be written as -``type(x)`` but ``type(x)`` is discouraged. +**Note**: `typeof(x)` can for historical reasons also be written as +`type(x)` but `type(x)` is discouraged. -One can obtain the type of a given expression by constructing a ``typeof`` +One can obtain the type of a given expression by constructing a `typeof` value from it (in many other languages this is known as the `typeof`:idx: operator): -.. code-block:: nim - + ```nim var x = 0 var y: typeof(x) # y has type int + ``` -If ``typeof`` is used to determine the result type of a proc/iterator/converter -call ``c(X)`` (where ``X`` stands for a possibly empty list of arguments), the -interpretation where ``c`` is an iterator is preferred over the +If `typeof` is used to determine the result type of a proc/iterator/converter +call `c(X)` (where `X` stands for a possibly empty list of arguments), the +interpretation, where `c` is an iterator, is preferred over the other interpretations, but this behavior can be changed by -passing ``typeOfProc`` as the second argument to ``typeof``: - -.. code-block:: nim - :test: "nim c $1" +passing `typeOfProc` as the second argument to `typeof`: + ```nim test = "nim c $1" iterator split(s: string): string = discard proc split(s: string): seq[string] = discard - # since an iterator is the preferred interpretation, `y` has the type ``string``: + # since an iterator is the preferred interpretation, this has the type `string`: assert typeof("a b c".split) is string assert typeof("a b c".split, typeOfProc) is seq[string] + ``` @@ -5644,34 +6812,35 @@ Modules Nim supports splitting a program into pieces by a module concept. Each module needs to be in its own file and has its own `namespace`:idx:. Modules enable `information hiding`:idx: and `separate compilation`:idx:. -A module may gain access to symbols of another module by the `import`:idx: -statement. `Recursive module dependencies`:idx: are allowed, but slightly -subtle. Only top-level symbols that are marked with an asterisk (``*``) are +A module may gain access to the symbols of another module by the `import`:idx: +statement. `Recursive module dependencies`:idx: are allowed, but are slightly +subtle. Only top-level symbols that are marked with an asterisk (`*`) are exported. A valid module name can only be a valid Nim identifier (and thus its filename is ``identifier.nim``). The algorithm for compiling modules is: -- compile the whole module as usual, following import statements recursively +- Compile the whole module as usual, following import statements recursively. -- if there is a cycle only import the already parsed symbols (that are - exported); if an unknown identifier occurs then abort +- If there is a cycle, only import the already parsed symbols (that are + exported); if an unknown identifier occurs then abort. This is best illustrated by an example: -.. code-block:: nim + ```nim # Module A type - T1* = int # Module A exports the type ``T1`` + T1* = int # Module A exports the type `T1` import B # the compiler starts parsing B proc main() = var i = p(3) # works because B has been parsed completely here main() + ``` -.. code-block:: nim + ```nim # Module B import A # A is not parsed here! Only the already known symbols # of A are imported. @@ -5680,170 +6849,184 @@ This is best illustrated by an example: # this works because the compiler has already # added T1 to A's interface symbol table result = x + 1 + ``` Import statement -~~~~~~~~~~~~~~~~ +---------------- -After the ``import`` statement a list of module names can follow or a single -module name followed by an ``except`` list to prevent some symbols to be +After the `import` keyword, a list of module names can follow or a single +module name followed by an `except` list to prevent some symbols from being imported: -.. code-block:: nim - :test: "nim c $1" - :status: 1 - - import strutils except `%`, toUpperAscii + ```nim test = "nim c $1" status = 1 + import std/strutils except `%`, toUpperAscii # doesn't work then: echo "$1" % "abc".toUpperAscii + ``` -It is not checked that the ``except`` list is really exported from the module. -This feature allows to compile against an older version of the module that -does not export these identifiers. +It is not checked that the `except` list is really exported from the module. +This feature allows us to compile against different versions of the module, +even when one version does not export some of these identifiers. -The ``import`` statement is only allowed at the top level. +The `import` statement is only allowed at the top level. +String literals can be used for import/include statements. +The compiler performs [path substitution](nimc.html#compiler-usage-commandminusline-switches) when used. Include statement -~~~~~~~~~~~~~~~~~ -The ``include`` statement does something fundamentally different than -importing a module: it merely includes the contents of a file. The ``include`` +----------------- + +The `include` statement does something fundamentally different than +importing a module: it merely includes the contents of a file. The `include` statement is useful to split up a large module into several files: -.. code-block:: nim + ```nim include fileA, fileB, fileC + ``` -The ``include`` statement can be used outside of the top level, as such: +The `include` statement can be used outside the top level, as such: -.. code-block:: nim + ```nim # Module A echo "Hello World!" + ``` -.. code-block:: nim + ```nim # Module B proc main() = include A main() # => Hello World! + ``` Module names in imports -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- -A module alias can be introduced via the ``as`` keyword: +A module alias can be introduced via the `as` keyword, after which the original module name +is inaccessible: -.. code-block:: nim - import strutils as su, sequtils as qu + ```nim + import std/strutils as su, std/sequtils as qu echo su.format("$1", "lalelu") + ``` -The original module name is then not accessible. The notations -``path/to/module`` or ``"path/to/module"`` can be used to refer to a module +The notations `path/to/module` or `"path/to/module"` can be used to refer to a module in subdirectories: -.. code-block:: nim + ```nim import lib/pure/os, "lib/pure/times" + ``` -Note that the module name is still ``strutils`` and not ``lib/pure/strutils`` -and so one **cannot** do: +Note that the module name is still `strutils` and not `lib/pure/strutils`, +thus one **cannot** do: -.. code-block:: nim + ```nim import lib/pure/strutils echo lib/pure/strutils.toUpperAscii("abc") + ``` -Likewise the following does not make sense as the name is ``strutils`` already: +Likewise, the following does not make sense as the name is `strutils` already: -.. code-block:: nim + ```nim import lib/pure/strutils as strutils + ``` Collective imports from a directory -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +----------------------------------- -The syntax ``import dir / [moduleA, moduleB]`` can be used to import multiple modules +The syntax `import dir / [moduleA, moduleB]` can be used to import multiple modules from the same directory. Path names are syntactically either Nim identifiers or string literals. If the path name is not a valid Nim identifier it needs to be a string literal: -.. code-block:: nim + ```nim import "gfx/3d/somemodule" # in quotes because '3d' is not a valid Nim identifier + ``` Pseudo import/include paths -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------------------- -A directory can also be a so called "pseudo directory". They can be used to +A directory can also be a so-called "pseudo directory". They can be used to avoid ambiguity when there are multiple modules with the same path. There are two pseudo directories: -1. ``std``: The ``std`` pseudo directory is the abstract location of Nim's standard -library. For example, the syntax ``import std / strutils`` is used to unambiguously -refer to the standard library's ``strutils`` module. -2. ``pkg``: The ``pkg`` pseudo directory is used to unambiguously refer to a Nimble -package. However, for technical details that lie outside of the scope of this document -its semantics are: *Use the search path to look for module name but ignore the standard -library locations*. In other words, it is the opposite of ``std``. +1. `std`: The `std` pseudo directory is the abstract location of Nim's standard + library. For example, the syntax `import std / strutils` is used to unambiguously + refer to the standard library's `strutils` module. +2. `pkg`: The `pkg` pseudo directory is used to unambiguously refer to a Nimble + package. However, for technical details that lie outside the scope of this document, + its semantics are: *Use the search path to look for module name but ignore the standard + library locations*. In other words, it is the opposite of `std`. +It is recommended and preferred but not currently enforced that all stdlib module imports include the std/ "pseudo directory" as part of the import name. From import statement -~~~~~~~~~~~~~~~~~~~~~ +--------------------- -After the ``from`` statement a module name follows followed by -an ``import`` to list the symbols one likes to use without explicit +After the `from` keyword, a module name followed by +an `import` to list the symbols one likes to use without explicit full qualification: -.. code-block:: nim - :test: "nim c $1" - - from strutils import `%` + ```nim test = "nim c $1" + from std/strutils import `%` echo "$1" % "abc" # always possible: full qualification: echo strutils.replace("abc", "a", "z") + ``` -It's also possible to use ``from module import nil`` if one wants to import +It's also possible to use `from module import nil` if one wants to import the module but wants to enforce fully qualified access to every symbol -in ``module``. +in `module`. Export statement -~~~~~~~~~~~~~~~~ +---------------- -An ``export`` statement can be used for symbol forwarding so that client +An `export` statement can be used for symbol forwarding so that client modules don't need to import a module's dependencies: -.. code-block:: nim + ```nim # module B type MyObject* = object + ``` -.. code-block:: nim + ```nim # module A import B export B.MyObject proc `$`*(x: MyObject): string = "my object" + ``` -.. code-block:: nim + ```nim # module C import A # B.MyObject has been imported implicitly here: var x: MyObject echo $x + ``` When the exported symbol is another module, all of its definitions will -be forwarded. One can use an ``except`` list to exclude some of the symbols. +be forwarded. One can use an `except` list to exclude some of the symbols. Notice that when exporting, one needs to specify only the module name: -.. code-block:: nim + ```nim import foo/bar/baz export baz + ``` @@ -5854,8 +7037,8 @@ the block in which the declaration occurred. The range where the identifier is known is the scope of the identifier. The exact scope of an identifier depends on the way it was declared. -Block scope -~~~~~~~~~~~ +### Block scope + The *scope* of a variable declared in the declaration part of a block is valid from the point of declaration until the end of the block. If a block contains a second block, in which the identifier is redeclared, @@ -5865,8 +7048,8 @@ identifier cannot be redefined in the same block, except if valid for procedure or iterator overloading purposes. -Tuple or object scope -~~~~~~~~~~~~~~~~~~~~~ +### Tuple or object scope + The field identifiers inside a tuple or object definition are valid in the following places: @@ -5874,32 +7057,99 @@ following places: * Field designators of a variable of the given tuple/object type. * In all descendant types of the object type. -Module scope -~~~~~~~~~~~~ +### Module scope + All identifiers of a module are valid from the point of declaration until the end of the module. Identifiers from indirectly dependent modules are *not* available. The `system`:idx: module is automatically imported in every module. -If a module imports an identifier by two different modules, each occurrence of -the identifier has to be qualified, unless it is an overloaded procedure or -iterator in which case the overloading resolution takes place: +If a module imports the same identifier from two different modules, the +identifier is considered ambiguous, which can be resolved in the following ways: + +* Qualifying the identifier as `module.identifier` resolves ambiguity + between modules. (See below for the case that the module name itself + is ambiguous.) +* Calling the identifier as a routine makes overload resolution take place, + which resolves ambiguity in the case that one overload matches stronger + than the others. +* Using the identifier in a context where the compiler can infer the type + of the identifier resolves ambiguity in the case that one definition + matches the type stronger than the others. -.. code-block:: nim + ```nim # Module A var x*: string + proc foo*(a: string) = + echo "A: ", a + ``` -.. code-block:: nim + ```nim # Module B var x*: int + proc foo*(b: int) = + echo "B: ", b + ``` -.. code-block:: nim + ```nim # Module C import A, B + + foo("abc") # A: abc + foo(123) # B: 123 + let inferred: proc (x: string) = foo + foo("def") # A: def + write(stdout, x) # error: x is ambiguous write(stdout, A.x) # no error: qualifier used + + proc bar(a: int): int = a + 1 + assert bar(x) == x + 1 # no error: only A.x of type int matches var x = 4 write(stdout, x) # not ambiguous: uses the module C's x + ``` +Modules can share their name, however, when trying to qualify an identifier with the module name the compiler will fail with ambiguous identifier error. One can qualify the identifier by aliasing the module. + + +```nim +# Module A/C +proc fb* = echo "fizz" +``` + + +```nim +# Module B/C +proc fb* = echo "buzz" +``` + + +```nim +import A/C +import B/C + +C.fb() # Error: ambiguous identifier: 'C' +``` + + +```nim +import A/C as fizz +import B/C + +fizz.fb() # Works +``` + + +Packages +-------- +A collection of modules in a file tree with an ``identifier.nimble`` file in the +root of the tree is called a Nimble package. A valid package name can only be a +valid Nim identifier and thus its filename is ``identifier.nimble`` where +``identifier`` is the desired package name. A module without a ``.nimble`` file +is assigned the package identifier: `unknown`. + +The distinction between packages allows diagnostic compiler messages to be +scoped to the current project's package vs foreign packages. + Compiler Messages @@ -5917,7 +7167,7 @@ Pragmas Pragmas are Nim's method to give the compiler additional information / commands without introducing a massive number of new keywords. Pragmas are processed on the fly during semantic checking. Pragmas are enclosed in the -special ``{.`` and ``.}`` curly brackets. Pragmas are also often used as a +special `{.` and `.}` curly brackets. Pragmas are also often used as a first implementation to play with a language feature before a nicer syntax to access the feature becomes available. @@ -5927,73 +7177,45 @@ deprecated pragma The deprecated pragma is used to mark a symbol as deprecated: -.. code-block:: nim + ```nim proc p() {.deprecated.} var x {.deprecated.}: char + ``` This pragma can also take in an optional warning string to relay to developers. -.. code-block:: nim + ```nim proc thing(x: bool) {.deprecated: "use thong instead".} + ``` -noSideEffect pragma -------------------- - -The ``noSideEffect`` pragma is used to mark a proc/iterator to have no side -effects. This means that the proc/iterator only changes locations that are -reachable from its parameters and the return value only depends on the -arguments. If none of its parameters have the type ``var T`` or ``out T`` -or ``ref T`` or ``ptr T`` this means no locations are modified. It is a static -error to mark a proc/iterator to have no side effect if the compiler cannot -verify this. - -As a special semantic rule, the built-in `debugEcho -<system.html#debugEcho,varargs[typed,]>`_ pretends to be free of side effects, -so that it can be used for debugging routines marked as ``noSideEffect``. - -``func`` is syntactic sugar for a proc with no side effects: - -.. code-block:: nim - func `+` (x, y: int): int - - -To override the compiler's side effect analysis a ``{.noSideEffect.}`` -pragma block can be used: - -.. code-block:: nim - - func f() = - {.noSideEffect.}: - echo "test" - compileTime pragma ------------------ -The ``compileTime`` pragma is used to mark a proc or variable to be used only +The `compileTime` pragma is used to mark a proc or variable to be used only during compile-time execution. No code will be generated for it. Compile-time procs are useful as helpers for macros. Since version 0.12.0 of the language, a -proc that uses ``system.NimNode`` within its parameter types is implicitly -declared ``compileTime``: +proc that uses `system.NimNode` within its parameter types is implicitly +declared `compileTime`: -.. code-block:: nim + ```nim proc astHelper(n: NimNode): NimNode = result = n + ``` Is the same as: -.. code-block:: nim + ```nim proc astHelper(n: NimNode): NimNode {.compileTime.} = result = n + ``` -``compileTime`` variables are available at runtime too. This simplifies certain +`compileTime` variables are available at runtime too. This simplifies certain idioms where variables are filled at compile-time (for example, lookup tables) but accessed at runtime: -.. code-block:: nim - :test: "nim c -r $1" - - import macros + ```nim test = "nim c -r $1" + import std/macros var nameToProc {.compileTime.}: seq[(string, proc (): string {.nimcall.})] @@ -6010,61 +7232,64 @@ but accessed at runtime: proc baz: string {.registerProc.} = "baz" doAssert nameToProc[2][1]() == "baz" + ``` -noReturn pragma +noreturn pragma --------------- -The ``noreturn`` pragma is used to mark a proc that never returns. +The `noreturn` pragma is used to mark a proc that never returns. acyclic pragma -------------- -The ``acyclic`` pragma can be used for object types to mark them as acyclic +The `acyclic` pragma can be used for object types to mark them as acyclic even though they seem to be cyclic. This is an **optimization** for the garbage collector to not consider objects of this type as part of a cycle: -.. code-block:: nim + ```nim type Node = ref NodeObj NodeObj {.acyclic.} = object left, right: Node data: string + ``` Or if we directly use a ref object: -.. code-block:: nim + ```nim type Node {.acyclic.} = ref object left, right: Node data: string + ``` -In the example a tree structure is declared with the ``Node`` type. Note that +In the example, a tree structure is declared with the `Node` type. Note that the type definition is recursive and the GC has to assume that objects of -this type may form a cyclic graph. The ``acyclic`` pragma passes the +this type may form a cyclic graph. The `acyclic` pragma passes the information that this cannot happen to the GC. If the programmer uses the -``acyclic`` pragma for data types that are in reality cyclic, the memory leaks -can be the result, but memory safety is preserved. +`acyclic` pragma for data types that are in reality cyclic, this may result +in memory leaks, but memory safety is preserved. final pragma ------------ -The ``final`` pragma can be used for an object type to specify that it +The `final` pragma can be used for an object type to specify that it cannot be inherited from. Note that inheritance is only available for -objects that inherit from an existing object (via the ``object of SuperType`` -syntax) or that have been marked as ``inheritable``. +objects that inherit from an existing object (via the `object of SuperType` +syntax) or that have been marked as `inheritable`. shallow pragma -------------- -The ``shallow`` pragma affects the semantics of a type: The compiler is +The `shallow` pragma affects the semantics of a type: The compiler is allowed to make a shallow copy. This can cause serious semantic issues and break memory safety! However, it can speed up assignments considerably, because the semantics of Nim require deep copying of sequences and strings. This can be expensive, especially if sequences are used to build a tree structure: -.. code-block:: nim + ```nim type NodeKind = enum nkLeaf, nkInner Node {.shallow.} = object @@ -6073,24 +7298,25 @@ structure: strVal: string of nkInner: children: seq[Node] + ``` pure pragma ----------- -An object type can be marked with the ``pure`` pragma so that its type field +An object type can be marked with the `pure` pragma so that its type field which is used for runtime type identification is omitted. This used to be necessary for binary compatibility with other compiled languages. -An enum type can be marked as ``pure``. Then access of its fields always +An enum type can be marked as `pure`. Then access of its fields always requires full qualification. asmNoStackFrame pragma ---------------------- -A proc can be marked with the ``asmNoStackFrame`` pragma to tell the compiler +A proc can be marked with the `asmNoStackFrame` pragma to tell the compiler it should not generate a stack frame for the proc. There are also no exit -statements like ``return result;`` generated and the generated C function is -declared as ``__declspec(naked)`` or ``__attribute__((naked))`` (depending on +statements like `return result;` generated and the generated C function is +declared as `__declspec(naked)`:c: or `__attribute__((naked))`:c: (depending on the used C compiler). **Note**: This pragma should only be used by procs which consist solely of @@ -6098,65 +7324,67 @@ assembler statements. error pragma ------------ -The ``error`` pragma is used to make the compiler output an error message -with the given content. Compilation does not necessarily abort after an error +The `error` pragma is used to make the compiler output an error message +with the given content. The compilation does not necessarily abort after an error though. -The ``error`` pragma can also be used to +The `error` pragma can also be used to annotate a symbol (like an iterator or proc). The *usage* of the symbol then triggers a static error. This is especially useful to rule out that some operation is valid due to overloading and type conversions: -.. code-block:: nim + ```nim ## check that underlying int values are compared and not the pointers: proc `==`(x, y: ptr int): bool {.error.} + ``` fatal pragma ------------ -The ``fatal`` pragma is used to make the compiler output an error message -with the given content. In contrast to the ``error`` pragma, compilation +The `fatal` pragma is used to make the compiler output an error message +with the given content. In contrast to the `error` pragma, the compilation is guaranteed to be aborted by this pragma. Example: -.. code-block:: nim + ```nim when not defined(objc): {.fatal: "Compile this program with the objc command!".} + ``` warning pragma -------------- -The ``warning`` pragma is used to make the compiler output a warning message +The `warning` pragma is used to make the compiler output a warning message with the given content. Compilation continues after the warning. hint pragma ----------- -The ``hint`` pragma is used to make the compiler output a hint message with +The `hint` pragma is used to make the compiler output a hint message with the given content. Compilation continues after the hint. line pragma ----------- -The ``line`` pragma can be used to affect line information of the annotated -statement as seen in stack backtraces: - -.. code-block:: nim +The `line` pragma can be used to affect line information of the annotated +statement, as seen in stack backtraces: + ```nim template myassert*(cond: untyped, msg = "") = if not cond: # change run-time line information of the 'raise' statement: {.line: instantiationInfo().}: - raise newException(EAssertionFailed, msg) + raise newException(AssertionDefect, msg) + ``` -If the ``line`` pragma is used with a parameter, the parameter needs be a -``tuple[filename: string, line: int]``. If it is used without a parameter, -``system.InstantiationInfo()`` is used. +If the `line` pragma is used with a parameter, the parameter needs to be a +`tuple[filename: string, line: int]`. If it is used without a parameter, +`system.instantiationInfo()` is used. linearScanEnd pragma -------------------- -The ``linearScanEnd`` pragma can be used to tell the compiler how to +The `linearScanEnd` pragma can be used to tell the compiler how to compile a Nim `case`:idx: statement. Syntactically it has to be used as a statement: -.. code-block:: nim + ```nim case myInt of 0: echo "most common case" @@ -6165,27 +7393,27 @@ statement: echo "second most common case" of 2: echo "unlikely: use branch table" else: echo "unlikely too: use branch table for ", myInt + ``` -In the example, the case branches ``0`` and ``1`` are much more common than -the other cases. Therefore the generated assembler code should test for these -values first, so that the CPU's branch predictor has a good chance to succeed +In the example, the case branches `0` and `1` are much more common than +the other cases. Therefore, the generated assembler code should test for these +values first so that the CPU's branch predictor has a good chance to succeed (avoiding an expensive CPU pipeline stall). The other cases might be put into a -jump table for O(1) overhead, but at the cost of a (very likely) pipeline +jump table for O(1) overhead but at the cost of a (very likely) pipeline stall. -The ``linearScanEnd`` pragma should be put into the last branch that should be +The `linearScanEnd` pragma should be put into the last branch that should be tested against via linear scanning. If put into the last branch of the -whole ``case`` statement, the whole ``case`` statement uses linear scanning. +whole `case` statement, the whole `case` statement uses linear scanning. computedGoto pragma ------------------- -The ``computedGoto`` pragma can be used to tell the compiler how to -compile a Nim `case`:idx: in a ``while true`` statement. +The `computedGoto` pragma can be used to tell the compiler how to +compile a Nim `case`:idx: in a `while true` statement. Syntactically it has to be used as a statement inside the loop: -.. code-block:: nim - + ```nim type MyEnum = enum enumA, enumB, enumC, enumD, enumE @@ -6217,8 +7445,9 @@ Syntactically it has to be used as a statement inside the loop: inc(pc) vm() + ``` -As the example shows ``computedGoto`` is mostly useful for interpreters. If +As the example shows, `computedGoto` is mostly useful for interpreters. If the underlying backend (C compiler) does not support the computed goto extension the pragma is simply ignored. @@ -6226,9 +7455,24 @@ extension the pragma is simply ignored. immediate pragma ---------------- -The immediate pragma is obsolete. See `Typed vs untyped parameters -<#templates-typed-vs-untyped-parameters>`_. +The immediate pragma is obsolete. See [Typed vs untyped parameters]. + +redefine pragma +--------------- + +Redefinition of template symbols with the same signature is allowed. +This can be made explicit with the `redefine` pragma: + +```nim +template foo: int = 1 +echo foo() # 1 +template foo: int {.redefine.} = 2 +echo foo() # 2 +# warning: implicit redefinition of template +template foo: int = 3 +``` +This is mostly intended for macro generated code. compilation option pragmas -------------------------- @@ -6266,9 +7510,10 @@ callconv cdecl|... Specifies the default calling convention for Example: -.. code-block:: nim + ```nim {.checks: off, optimization: speed.} # compile without runtime checks and optimize for speed + ``` push and pop pragmas @@ -6276,16 +7521,17 @@ push and pop pragmas The `push/pop`:idx: pragmas are very similar to the option directive, but are used to override the settings temporarily. Example: -.. code-block:: nim + ```nim {.push checks: off.} # compile this section without runtime checks as it is # speed critical # ... some code ... {.pop.} # restore old settings + ``` `push/pop`:idx: can switch on/off some standard library pragmas, example: -.. code-block:: nim + ```nim {.push inline.} proc thisIsInlined(): int = 42 func willBeInlined(): float = 42.0 @@ -6296,34 +7542,36 @@ but are used to override the settings temporarily. Example: template example(): string = "https://nim-lang.org" {.pop.} - {.push deprecated, hint[LineTooLong]: off, used, stackTrace: off.} + {.push deprecated, used, stackTrace: off.} proc sample(): bool = true {.pop.} + ``` -For third party pragmas it depends on its implementation, but uses the same syntax. +For third party pragmas, it depends on its implementation but uses the same syntax. register pragma --------------- -The ``register`` pragma is for variables only. It declares the variable as -``register``, giving the compiler a hint that the variable should be placed +The `register` pragma is for variables only. It declares the variable as +`register`, giving the compiler a hint that the variable should be placed in a hardware register for faster access. C compilers usually ignore this though and for good reasons: Often they do a better job without it anyway. -In highly specific cases (a dispatch loop of a bytecode interpreter for -example) it may provide benefits, though. +However, in highly specific cases (a dispatch loop of a bytecode interpreter +for example) it may provide benefits. global pragma ------------- -The ``global`` pragma can be applied to a variable within a proc to instruct +The `global` pragma can be applied to a variable within a proc to instruct the compiler to store it in a global location and initialize it once at program startup. -.. code-block:: nim + ```nim proc isHexNumber(s: string): bool = var pattern {.global.} = re"[0-9a-fA-F]+" result = s.match(pattern) + ``` When used within a generic proc, a separate unique global variable will be created for each instantiation of the proc. The order of initialization of @@ -6333,13 +7581,15 @@ and before any variable in a module that imports it. Disabling certain messages -------------------------- -Nim generates some warnings and hints ("line too long") that may annoy the +Nim generates some warnings and hints that may annoy the user. A mechanism for disabling certain messages is provided: Each hint -and warning message contains a symbol in brackets. This is the message's -identifier that can be used to enable or disable it: +and warning message is associated with a symbol. This is the message's +identifier, which can be used to enable or disable the message by putting it +in brackets following the pragma: -.. code-block:: Nim - {.hint[LineTooLong]: off.} # turn off the hint about too long lines + ```Nim + {.hint[XDeclaredButNotUsed]: off.} # Turn off the hint about declared but not used symbols. + ``` This is often better than disabling all warnings at once. @@ -6348,10 +7598,10 @@ used pragma ----------- Nim produces a warning for symbols that are not exported and not used either. -The ``used`` pragma can be attached to a symbol to suppress this warning. This +The `used` pragma can be attached to a symbol to suppress this warning. This is particularly useful when the symbol was generated by a macro: -.. code-block:: nim + ```nim template implementArithOps(T) = proc echoAdd(a, b: T) {.used.} = echo a + b @@ -6361,32 +7611,34 @@ is particularly useful when the symbol was generated by a macro: # no warning produced for the unused 'echoSub' implementArithOps(int) echoAdd 3, 5 + ``` -``used`` can also be used as a top level statement to mark a module as "used". +`used` can also be used as a top-level statement to mark a module as "used". This prevents the "Unused import" warning: -.. code-block:: nim - + ```nim # module: debughelper.nim when defined(nimHasUsed): # 'import debughelper' is so useful for debugging # that Nim shouldn't produce a warning for that import, # even if currently unused: {.used.} + ``` experimental pragma ------------------- -The ``experimental`` pragma enables experimental language features. Depending -on the concrete feature this means that the feature is either considered +The `experimental` pragma enables experimental language features. Depending +on the concrete feature, this means that the feature is either considered too unstable for an otherwise stable release or that the future of the feature -is uncertain (it may be removed any time). +is uncertain (it may be removed at any time). See the +[experimental manual](manual_experimental.html) for more details. Example: -.. code-block:: nim - import threadpool + ```nim + import std/threadpool {.experimental: "parallel".} proc threadedEcho(s: string, i: int) = @@ -6398,15 +7650,15 @@ Example: spawn threadedEcho("echo in parallel", i) useParallel() + ``` -As a top level statement, the experimental pragma enables a feature for the +As a top-level statement, the experimental pragma enables a feature for the rest of the module it's enabled in. This is problematic for macro and generic -instantiations that cross a module scope. Currently these usages have to be -put into a ``.push/pop`` environment: - -.. code-block:: nim +instantiations that cross a module scope. Currently, these usages have to be +put into a `.push/pop` environment: + ```nim # client.nim proc useParallel*[T](unused: T) = # use a generic T here to show the problem. @@ -6416,12 +7668,13 @@ put into a ``.push/pop`` environment: echo "echo in parallel" {.pop.} + ``` -.. code-block:: nim - + ```nim import client useParallel(1) + ``` Implementation Specific Pragmas @@ -6433,20 +7686,53 @@ supports but which should not be seen as part of the language specification. Bitsize pragma -------------- -The ``bitsize`` pragma is for object field members. It declares the field as +The `bitsize` pragma is for object field members. It declares the field as a bitfield in C/C++. -.. code-block:: Nim + ```Nim type mybitfield = object flag {.bitsize:1.}: cuint + ``` generates: -.. code-block:: C + ```C struct mybitfield { unsigned int flag:1; }; + ``` + + +size pragma +----------- +Nim automatically determines the size of an enum. +But when wrapping a C enum type, it needs to be of a specific size. +The `size pragma` allows specifying the size of the enum type. + + ```Nim + type + EventType* {.size: sizeof(uint32).} = enum + QuitEvent, + AppTerminating, + AppLowMemory + + doAssert sizeof(EventType) == sizeof(uint32) + ``` + +The `size pragma` can also specify the size of an `importc` incomplete object type +so that one can get the size of it at compile time even if it was declared without fields. + + ```Nim + type + AtomicFlag* {.importc: "atomic_flag", header: "<stdatomic.h>", size: 1.} = object + + static: + # if AtomicFlag didn't have the size pragma, this code would result in a compile time error. + echo sizeof(AtomicFlag) + ``` + +The `size pragma` accepts only the values 1, 2, 4 or 8. Align pragma @@ -6456,163 +7742,203 @@ The `align`:idx: pragma is for variables and object field members. It modifies the alignment requirement of the entity being declared. The argument must be a constant power of 2. Valid non-zero alignments that are weaker than other align pragmas on the same -declaration are ignored. Alignments that are weaker that the +declaration are ignored. Alignments that are weaker than the alignment requirement of the type are ignored. -.. code-block:: Nim + ```Nim + type + sseType = object + sseData {.align(16).}: array[4, float32] - type - sseType = object - sseData {.align(16).}: array[4, float32] + # every object will be aligned to 128-byte boundary + Data = object + x: char + cacheline {.align(128).}: array[128, char] # over-aligned array of char, - # every object will be aligned to 128-byte boundary - Data = object - x: char - cacheline {.align(128).}: array[128, char] # over-aligned array of char, + proc main() = + echo "sizeof(Data) = ", sizeof(Data), " (1 byte + 127 bytes padding + 128-byte array)" + # output: sizeof(Data) = 256 (1 byte + 127 bytes padding + 128-byte array) + echo "alignment of sseType is ", alignof(sseType) + # output: alignment of sseType is 16 + var d {.align(2048).}: Data # this instance of data is aligned even stricter - proc main() = - echo "sizeof(Data) = ", sizeof(Data), " (1 byte + 127 bytes padding + 128-byte array)" - # output: sizeof(Data) = 256 (1 byte + 127 bytes padding + 128-byte array) - echo "alignment of sseType is ", alignof(sseType) - # output: alignment of sseType is 16 - var d {.align(2048).}: Data # this instance of data is aligned even stricter + main() + ``` + +This pragma has no effect on the JS backend. + + +Noalias pragma +-------------- - main() +Since version 1.4 of the Nim compiler, there is a `.noalias` annotation for variables +and parameters. It is mapped directly to C/C++'s `restrict`:c: keyword and means that +the underlying pointer is pointing to a unique location in memory, no other aliases to +this location exist. It is *unchecked* that this alias restriction is followed. If the +restriction is violated, the backend optimizer is free to miscompile the code. +This is an **unsafe** language feature. -This pragma has no effect for the JS backend. +Ideally in later versions of the language, the restriction will be enforced at +compile time. (This is also why the name `noalias` was chosen instead of a more +verbose name like `unsafeAssumeNoAlias`.) Volatile pragma --------------- -The ``volatile`` pragma is for variables only. It declares the variable as -``volatile``, whatever that means in C/C++ (its semantics are not well defined +The `volatile` pragma is for variables only. It declares the variable as +`volatile`:c:, whatever that means in C/C++ (its semantics are not well-defined in C/C++). **Note**: This pragma will not exist for the LLVM backend. -NoDecl pragma +nodecl pragma ------------- -The ``noDecl`` pragma can be applied to almost any symbol (variable, proc, +The `nodecl` pragma can be applied to almost any symbol (variable, proc, type, etc.) and is sometimes useful for interoperability with C: It tells Nim that it should not generate a declaration for the symbol in the C code. For example: -.. code-block:: Nim + ```Nim var - EACCES {.importc, noDecl.}: cint # pretend EACCES was a variable, as + EACCES {.importc, nodecl.}: cint # pretend EACCES was a variable, as # Nim does not know its value + ``` -However, the ``header`` pragma is often the better alternative. +However, the `header` pragma is often the better alternative. **Note**: This will not work for the LLVM backend. Header pragma ------------- -The ``header`` pragma is very similar to the ``noDecl`` pragma: It can be +The `header` pragma is very similar to the `nodecl` pragma: It can be applied to almost any symbol and specifies that it should not be declared -and instead the generated code should contain an ``#include``: +and instead, the generated code should contain an `#include`:c:\: -.. code-block:: Nim + ```Nim type PFile {.importc: "FILE*", header: "<stdio.h>".} = distinct pointer # import C's FILE* type; Nim will treat it as a new pointer type + ``` -The ``header`` pragma always expects a string constant. The string constant +The `header` pragma always expects a string constant. The string constant contains the header file: As usual for C, a system header file is enclosed -in angle brackets: ``<>``. If no angle brackets are given, Nim -encloses the header file in ``""`` in the generated C code. +in angle brackets: `<>`:c:. If no angle brackets are given, Nim +encloses the header file in `""`:c: in the generated C code. **Note**: This will not work for the LLVM backend. IncompleteStruct pragma ----------------------- -The ``incompleteStruct`` pragma tells the compiler to not use the -underlying C ``struct`` in a ``sizeof`` expression: +The `incompleteStruct` pragma tells the compiler to not use the +underlying C `struct`:c: in a `sizeof` expression: -.. code-block:: Nim + ```Nim type DIR* {.importc: "DIR", header: "<dirent.h>", pure, incompleteStruct.} = object + ``` Compile pragma -------------- -The ``compile`` pragma can be used to compile and link a C/C++ source file +The `compile` pragma can be used to compile and link a C/C++ source file with the project: -.. code-block:: Nim +This pragma can take three forms. The first is a simple file input: + ```Nim {.compile: "myfile.cpp".} + ``` + +The second form is a tuple where the second arg is the output name strutils formatter: + ```Nim + {.compile: ("file.c", "$1.o").} + ``` **Note**: Nim computes a SHA1 checksum and only recompiles the file if it -has changed. One can use the ``-f`` command line option to force recompilation -of the file. +has changed. One can use the `-f`:option: command-line option to force +the recompilation of the file. + +Since 1.4 the `compile` pragma is also available with this syntax: + + ```Nim + {.compile("myfile.cpp", "--custom flags here").} + ``` + +As can be seen in the example, this new variant allows for custom flags +that are passed to the C compiler when the file is recompiled. Link pragma ----------- -The ``link`` pragma can be used to link an additional file with the project: +The `link` pragma can be used to link an additional file with the project: -.. code-block:: Nim + ```Nim {.link: "myfile.o".} + ``` -PassC pragma +passc pragma ------------ -The ``passc`` pragma can be used to pass additional parameters to the C -compiler like one would using the commandline switch ``--passc``: +The `passc` pragma can be used to pass additional parameters to the C +compiler like one would use the command-line switch `--passc`:option:\: -.. code-block:: Nim + ```Nim {.passc: "-Wall -Werror".} + ``` -Note that one can use ``gorge`` from the `system module <system.html>`_ to +Note that one can use `gorge` from the [system module](system.html) to embed parameters from an external command that will be executed during semantic analysis: -.. code-block:: Nim + ```Nim {.passc: gorge("pkg-config --cflags sdl").} + ``` -LocalPassc pragma +localPassC pragma ----------------- -The ``localPassc`` pragma can be used to pass additional parameters to the C +The `localPassC` pragma can be used to pass additional parameters to the C compiler, but only for the C/C++ file that is produced from the Nim module the pragma resides in: -.. code-block:: Nim + ```Nim # Module A.nim # Produces: A.nim.cpp - {.localPassc: "-Wall -Werror".} # Passed when compiling A.nim.cpp + {.localPassC: "-Wall -Werror".} # Passed when compiling A.nim.cpp + ``` -PassL pragma +passl pragma ------------ -The ``passL`` pragma can be used to pass additional parameters to the linker -like one would using the commandline switch ``--passL``: +The `passl` pragma can be used to pass additional parameters to the linker +like one would be using the command-line switch `--passl`:option:\: -.. code-block:: Nim - {.passL: "-lSDLmain -lSDL".} + ```Nim + {.passl: "-lSDLmain -lSDL".} + ``` -Note that one can use ``gorge`` from the `system module <system.html>`_ to +Note that one can use `gorge` from the [system module](system.html) to embed parameters from an external command that will be executed during semantic analysis: -.. code-block:: Nim - {.passL: gorge("pkg-config --libs sdl").} + ```Nim + {.passl: gorge("pkg-config --libs sdl").} + ``` Emit pragma ----------- -The ``emit`` pragma can be used to directly affect the output of the +The `emit` pragma can be used to directly affect the output of the compiler's code generator. The code is then unportable to other code generators/backends. Its usage is highly discouraged! However, it can be extremely useful for interfacing with `C++`:idx: or `Objective C`:idx: code. Example: -.. code-block:: Nim + ```Nim {.emit: """ static int cvariable = 420; """.} @@ -6625,27 +7951,29 @@ Example: {.pop.} embedsC() + ``` -``nimbase.h`` defines ``NIM_EXTERNC`` C macro that can be used for -``extern "C"`` code to work with both ``nim c`` and ``nim cpp``, eg: +``nimbase.h`` defines `NIM_EXTERNC`:c: C macro that can be used for +`extern "C"`:cpp: code to work with both `nim c`:cmd: and `nim cpp`:cmd:, e.g.: -.. code-block:: Nim + ```Nim proc foobar() {.importc:"$1".} {.emit: """ #include <stdio.h> NIM_EXTERNC void fun(){} """.} + ``` -For backwards compatibility, if the argument to the ``emit`` statement -is a single string literal, Nim symbols can be referred to via backticks. -This usage is however deprecated. +.. note:: For backward compatibility, if the argument to the `emit` statement + is a single string literal, Nim symbols can be referred to via backticks. + This usage is however deprecated. -For a toplevel emit statement the section where in the generated C/C++ file -the code should be emitted can be influenced via the -prefixes ``/*TYPESECTION*/`` or ``/*VARSECTION*/`` or ``/*INCLUDESECTION*/``: +For a top-level emit statement, the section where in the generated C/C++ file +the code should be emitted can be influenced via the prefixes +`/*TYPESECTION*/`:c: or `/*VARSECTION*/`:c: or `/*INCLUDESECTION*/`:c:\: -.. code-block:: Nim + ```Nim {.emit: """/*TYPESECTION*/ struct Vector3 { public: @@ -6659,24 +7987,25 @@ prefixes ``/*TYPESECTION*/`` or ``/*VARSECTION*/`` or ``/*INCLUDESECTION*/``: x: cfloat proc constructVector3(a: cfloat): Vector3 {.importcpp: "Vector3(@)", nodecl} + ``` ImportCpp pragma ---------------- -**Note**: `c2nim <https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst>`_ can parse a large subset of C++ and knows -about the ``importcpp`` pragma pattern language. It is not necessary +**Note**: [c2nim](https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst) +can parse a large subset of C++ and knows +about the `importcpp` pragma pattern language. It is not necessary to know all the details described here. -Similar to the `importc pragma for C -<#foreign-function-interface-importc-pragma>`_, the -``importcpp`` pragma can be used to import `C++`:idx: methods or C++ symbols +Similar to the [importc pragma] for C, the +`importcpp` pragma can be used to import `C++`:idx: methods or C++ symbols in general. The generated code then uses the C++ method calling -syntax: ``obj->method(arg)``. In combination with the ``header`` and ``emit`` +syntax: `obj->method(arg)`:cpp:. In combination with the `header` and `emit` pragmas this allows *sloppy* interfacing with libraries written in C++: -.. code-block:: Nim + ```Nim # Horrible example of how to interface with a C++ engine ... ;-) {.link: "/usr/lib/libIrrlicht.so".} @@ -6702,215 +8031,224 @@ pragmas this allows *sloppy* interfacing with libraries written in C++: header: irr, importcpp: "createDevice(@)".} proc run(device: IrrlichtDevice): bool {. header: irr, importcpp: "#.run(@)".} + ``` -The compiler needs to be told to generate C++ (command ``cpp``) for -this to work. The conditional symbol ``cpp`` is defined when the compiler +The compiler needs to be told to generate C++ (command `cpp`:option:) for +this to work. The conditional symbol `cpp` is defined when the compiler emits C++ code. -Namespaces -~~~~~~~~~~ +### Namespaces -The *sloppy interfacing* example uses ``.emit`` to produce ``using namespace`` +The *sloppy interfacing* example uses `.emit` to produce `using namespace`:cpp: declarations. It is usually much better to instead refer to the imported name -via the ``namespace::identifier`` notation: +via the `namespace::identifier`:cpp: notation: -.. code-block:: nim + ```nim type IrrlichtDeviceObj {.header: irr, importcpp: "irr::IrrlichtDevice".} = object + ``` -Importcpp for enums -~~~~~~~~~~~~~~~~~~~ +### Importcpp for enums -When ``importcpp`` is applied to an enum type the numerical enum values are -annotated with the C++ enum type, like in this example: ``((TheCppEnum)(3))``. +When `importcpp` is applied to an enum type the numerical enum values are +annotated with the C++ enum type, like in this example: +`((TheCppEnum)(3))`:cpp:. (This turned out to be the simplest way to implement it.) -Importcpp for procs -~~~~~~~~~~~~~~~~~~~ +### Importcpp for procs -Note that the ``importcpp`` variant for procs uses a somewhat cryptic pattern +Note that the `importcpp` variant for procs uses a somewhat cryptic pattern language for maximum flexibility: - A hash ``#`` symbol is replaced by the first or next argument. - A dot following the hash ``#.`` indicates that the call should use C++'s dot or arrow notation. -- An at symbol ``@`` is replaced by the remaining arguments, separated by - commas. +- An at symbol ``@`` is replaced by the remaining arguments, + separated by commas. For example: -.. code-block:: nim + ```nim proc cppMethod(this: CppObj, a, b, c: cint) {.importcpp: "#.CppMethod(@)".} var x: ptr CppObj cppMethod(x[], 1, 2, 3) + ``` Produces: -.. code-block:: C + ```C x->CppMethod(1, 2, 3) + ``` -As a special rule to keep backwards compatibility with older versions of the -``importcpp`` pragma, if there is no special pattern +As a special rule to keep backward compatibility with older versions of the +`importcpp` pragma, if there is no special pattern character (any of ``# ' @``) at all, C++'s dot or arrow notation is assumed, so the above example can also be written as: -.. code-block:: nim + ```nim proc cppMethod(this: CppObj, a, b, c: cint) {.importcpp: "CppMethod".} + ``` Note that the pattern language naturally also covers C++'s operator overloading capabilities: -.. code-block:: nim + ```nim proc vectorAddition(a, b: Vec3): Vec3 {.importcpp: "# + #".} proc dictLookup(a: Dict, k: Key): Value {.importcpp: "#[#]".} + ``` - An apostrophe ``'`` followed by an integer ``i`` in the range 0..9 is replaced by the i'th parameter *type*. The 0th position is the result type. This can be used to pass types to C++ function templates. Between - the ``'`` and the digit an asterisk can be used to get to the base type - of the type. (So it "takes away a star" from the type; ``T*`` becomes ``T``.) + the ``'`` and the digit, an asterisk can be used to get to the base type + of the type. (So it "takes away a star" from the type; `T*`:c: becomes `T`.) Two stars can be used to get to the element type of the element type etc. For example: -.. code-block:: nim - + ```nim type Input {.importcpp: "System::Input".} = object proc getSubsystem*[T](): ptr T {.importcpp: "SystemManager::getSubsystem<'*0>()", nodecl.} let x: ptr Input = getSubsystem[Input]() + ``` Produces: -.. code-block:: C + ```C x = SystemManager::getSubsystem<System::Input>() + ``` -- ``#@`` is a special case to support a ``cnew`` operation. It is required so +- ``#@`` is a special case to support a `cnew` operation. It is required so that the call expression is inlined directly, without going through a temporary location. This is only required to circumvent a limitation of the current code generator. -For example C++'s ``new`` operator can be "imported" like this: +For example C++'s `new`:cpp: operator can be "imported" like this: -.. code-block:: nim + ```nim proc cnew*[T](x: T): ptr T {.importcpp: "(new '*0#@)", nodecl.} # constructor of 'Foo': proc constructFoo(a, b: cint): Foo {.importcpp: "Foo(@)".} let x = cnew constructFoo(3, 4) + ``` Produces: -.. code-block:: C + ```C x = new Foo(3, 4) + ``` -However, depending on the use case ``new Foo`` can also be wrapped like this +However, depending on the use case `new Foo`:cpp: can also be wrapped like this instead: -.. code-block:: nim + ```nim proc newFoo(a, b: cint): ptr Foo {.importcpp: "new Foo(@)".} let x = newFoo(3, 4) + ``` -Wrapping constructors -~~~~~~~~~~~~~~~~~~~~~ +### Wrapping constructors Sometimes a C++ class has a private copy constructor and so code like -``Class c = Class(1,2);`` must not be generated but instead ``Class c(1,2);``. +`Class c = Class(1,2);`:cpp: must not be generated but instead +`Class c(1,2);`:cpp:. For this purpose the Nim proc that wraps a C++ constructor needs to be annotated with the `constructor`:idx: pragma. This pragma also helps to generate faster C++ code since construction then doesn't invoke the copy constructor: -.. code-block:: nim + ```nim # a better constructor of 'Foo': proc constructFoo(a, b: cint): Foo {.importcpp: "Foo(@)", constructor.} + ``` -Wrapping destructors -~~~~~~~~~~~~~~~~~~~~ +### Wrapping destructors Since Nim generates C++ directly, any destructor is called implicitly by the C++ compiler at the scope exits. This means that often one can get away with -not wrapping the destructor at all! However when it needs to be invoked +not wrapping the destructor at all! However, when it needs to be invoked explicitly, it needs to be wrapped. The pattern language provides everything that is required: -.. code-block:: nim + ```nim proc destroyFoo(this: var Foo) {.importcpp: "#.~Foo()".} + ``` -Importcpp for objects -~~~~~~~~~~~~~~~~~~~~~ +### Importcpp for objects -Generic ``importcpp``'ed objects are mapped to C++ templates. This means that +Generic `importcpp`'ed objects are mapped to C++ templates. This means that one can import C++'s templates rather easily without the need for a pattern language for object types: -.. code-block:: nim + ```nim test = "nim cpp $1" type - StdMap {.importcpp: "std::map", header: "<map>".} [K, V] = object + StdMap[K, V] {.importcpp: "std::map", header: "<map>".} = object proc `[]=`[K, V](this: var StdMap[K, V]; key: K; val: V) {. importcpp: "#[#] = #", header: "<map>".} var x: StdMap[cint, cdouble] x[6] = 91.4 + ``` Produces: -.. code-block:: C + ```C std::map<int, double> x; x[6] = 91.4; + ``` -- If more precise control is needed, the apostrophe ``'`` can be used in the +- If more precise control is needed, the apostrophe `'` can be used in the supplied pattern to denote the concrete type parameters of the generic type. See the usage of the apostrophe operator in proc patterns for more details. -.. code-block:: nim - - type - VectorIterator {.importcpp: "std::vector<'0>::iterator".} [T] = object - - var x: VectorIterator[cint] + ```nim + type + VectorIterator[T] {.importcpp: "std::vector<'0>::iterator".} = object + var x: VectorIterator[cint] + ``` -Produces: + Produces: -.. code-block:: C + ```C - std::vector<int>::iterator x; + std::vector<int>::iterator x; + ``` ImportJs pragma --------------- -Similar to the `importcpp pragma for C++ <#implementation-specific-pragmas-importcpp-pragma>`_, -the ``importjs`` pragma can be used to import Javascript methods or +Similar to the [importcpp pragma] for C++, +the `importjs` pragma can be used to import Javascript methods or symbols in general. The generated code then uses the Javascript method calling syntax: ``obj.method(arg)``. ImportObjC pragma ----------------- -Similar to the `importc pragma for C -<#foreign-function-interface-importc-pragma>`_, the ``importobjc`` pragma can -be used to import `Objective C`:idx: methods. The generated code then uses the +Similar to the [importc pragma] for C, the `importobjc` pragma can +be used to import `Objective C`:idx: methods. The generated code then uses the Objective C method calling syntax: ``[obj method param1: arg]``. -In addition with the ``header`` and ``emit`` pragmas this +In addition with the `header` and `emit` pragmas this allows *sloppy* interfacing with libraries written in Objective C: -.. code-block:: Nim + ```Nim # horrible example of how to interface with GNUStep ... - {.passL: "-lobjc".} + {.passl: "-lobjc".} {.emit: """ #include <objc/Object.h> @interface Greeter:Object @@ -6942,8 +8280,9 @@ allows *sloppy* interfacing with libraries written in Objective C: var g = newGreeter() g.greet(12, 34) g.free() + ``` -The compiler needs to be told to generate Objective C (command ``objc``) for +The compiler needs to be told to generate Objective C (command `objc`:option:) for this to work. The conditional symbol ``objc`` is defined when the compiler emits Objective C code. @@ -6951,56 +8290,88 @@ emits Objective C code. CodegenDecl pragma ------------------ -The ``codegenDecl`` pragma can be used to directly influence Nim's code -generator. It receives a format string that determines how the variable -or proc is declared in the generated code. +The `codegenDecl` pragma can be used to directly influence Nim's code +generator. It receives a format string that determines how the variable, +proc or object type is declared in the generated code. -For variables $1 in the format string represents the type of the variable -and $2 is the name of the variable. +For variables, $1 in the format string represents the type of the variable, +$2 is the name of the variable, and each appearance of $# represents $1/$2 +respectively according to its position. The following Nim code: -.. code-block:: nim + ```nim var a {.codegenDecl: "$# progmem $#".}: int + ``` will generate this C code: -.. code-block:: c + ```c int progmem a + ``` -For procedures $1 is the return type of the procedure, $2 is the name of -the procedure and $3 is the parameter list. +For procedures, $1 is the return type of the procedure, $2 is the name of +the procedure, $3 is the parameter list, and each appearance of $# represents +$1/$2/$3 respectively according to its position. The following nim code: -.. code-block:: nim + ```nim proc myinterrupt() {.codegenDecl: "__interrupt $# $#$#".} = echo "realistic interrupt handler" + ``` will generate this code: -.. code-block:: c + ```c __interrupt void myinterrupt() + ``` +For object types, the $1 represents the name of the object type, $2 is the list of +fields and $3 is the base type. -InjectStmt pragma ------------------ +```nim + +const strTemplate = """ + struct $1 { + $2 + }; +""" +type Foo {.codegenDecl:strTemplate.} = object + a, b: int +``` + +will generate this code: -The ``injectStmt`` pragma can be used to inject a statement before every -other statement in the current module. It is only supposed to be used for -debugging: -.. code-block:: nim - {.injectStmt: gcInvariants().} +```c +struct Foo { + NI a; + NI b; +}; +``` - # ... complex code here that produces crashes ... +`cppNonPod` pragma +------------------ + +The `cppNonPod` pragma should be used for non-POD `importcpp` types so that they +work properly (in particular regarding constructor and destructor) for +`threadvar` variables. This requires `--tlsEmulation:off`:option:. + + ```nim + type Foo {.cppNonPod, importcpp, header: "funs.h".} = object + x: cint + proc main()= + var a {.threadvar.}: Foo + ``` -compile time define pragmas + +compile-time define pragmas --------------------------- The pragmas listed here can be used to optionally accept values from -the -d/--define option at compile time. +the `-d/--define`:option: option at compile time. The implementation currently provides the following possible options (various others may be added later). @@ -7013,19 +8384,40 @@ pragma description `booldefine`:idx: Reads in a build-time define as a bool ================= ============================================ -.. code-block:: nim - const FooBar {.intdefine.}: int = 5 - echo FooBar + ```nim + const FooBar {.intdefine.}: int = 5 + echo FooBar + ``` -:: - nim c -d:FooBar=42 foobar.nim + ```cmd + nim c -d:FooBar=42 foobar.nim + ``` -In the above example, providing the -d flag causes the symbol -``FooBar`` to be overwritten at compile time, printing out 42. If the -``-d:FooBar=42`` were to be omitted, the default value of 5 would be +In the above example, providing the `-d`:option: flag causes the symbol +`FooBar` to be overwritten at compile-time, printing out 42. If the +`-d:FooBar=42`:option: were to be omitted, the default value of 5 would be used. To see if a value was provided, `defined(FooBar)` can be used. -The syntax `-d:flag` is actually just a shortcut for `-d:flag=true`. +The syntax `-d:flag`:option: is actually just a shortcut for +`-d:flag=true`:option:. + +These pragmas also accept an optional string argument for qualified +define names. + + ```nim + const FooBar {.intdefine: "package.FooBar".}: int = 5 + echo FooBar + ``` + + ```cmd + nim c -d:package.FooBar=42 foobar.nim + ``` + +This helps disambiguate define names in different packages. + +See also the [generic `define` pragma](manual_experimental.html#generic-nimdefine-pragma) +for a version of these pragmas that detects the type of the define based on +the constant value. User-defined pragmas ==================== @@ -7034,43 +8426,46 @@ User-defined pragmas pragma pragma ------------- -The ``pragma`` pragma can be used to declare user defined pragmas. This is -useful because Nim's templates and macros do not affect pragmas. User -defined pragmas are in a different module-wide scope than all other symbols. +The `pragma` pragma can be used to declare user-defined pragmas. This is +useful because Nim's templates and macros do not affect pragmas. +User-defined pragmas are in a different module-wide scope than all other symbols. They cannot be imported from a module. Example: -.. code-block:: nim + ```nim when appType == "lib": {.pragma: rtl, exportc, dynlib, cdecl.} else: {.pragma: rtl, importc, dynlib: "client.dll", cdecl.} proc p*(a, b: int): int {.rtl.} = - result = a+b + result = a + b + ``` -In the example a new pragma named ``rtl`` is introduced that either imports +In the example, a new pragma named `rtl` is introduced that either imports a symbol from a dynamic library or exports the symbol for dynamic library generation. Custom annotations ------------------ -It is possible to define custom typed pragmas. Custom pragmas do not effect +It is possible to define custom typed pragmas. Custom pragmas do not affect code generation directly, but their presence can be detected by macros. -Custom pragmas are defined using templates annotated with pragma ``pragma``: +Custom pragmas are defined using templates annotated with pragma `pragma`: -.. code-block:: nim + ```nim template dbTable(name: string, table_space: string = "") {.pragma.} template dbKey(name: string = "", primary_key: bool = false) {.pragma.} template dbForeignKey(t: typedesc) {.pragma.} template dbIgnore {.pragma.} + ``` -Consider stylized example of possible Object Relation Mapping (ORM) implementation: +Consider this stylized example of a possible Object Relation Mapping (ORM) +implementation: -.. code-block:: nim + ```nim const tblspace {.strdefine.} = "dev" # switch for dev, test and prod environments type @@ -7085,9 +8480,10 @@ Consider stylized example of possible Object Relation Mapping (ORM) implementati user_id {.dbForeignKey: User.}: int read_access: bool write_access: bool - admin_acess: bool + admin_access: bool + ``` -In this example custom pragmas are used to describe how Nim objects are +In this example, custom pragmas are used to describe how Nim objects are mapped to the schema of the relational database. Custom pragmas can have zero or more arguments. In order to pass multiple arguments use one of template call syntaxes. All arguments are typed and follow standard @@ -7098,74 +8494,78 @@ Custom pragmas can be used in all locations where ordinary pragmas can be specified. It is possible to annotate procs, templates, type and variable definitions, statements, etc. -Macros module includes helpers which can be used to simplify custom pragma -access `hasCustomPragma`, `getCustomPragmaVal`. Please consult the macros module -documentation for details. These macros are not magic, everything they do can -also be achieved by walking the AST of the object representation. +The macros module includes helpers which can be used to simplify custom pragma +access `hasCustomPragma`, `getCustomPragmaVal`. Please consult the +[macros](macros.html) module documentation for details. These macros are not +magic, everything they do can also be achieved by walking the AST of the object +representation. More examples with custom pragmas: - Better serialization/deserialization control: -.. code-block:: nim - type MyObj = object - a {.dontSerialize.}: int - b {.defaultDeserialize: 5.}: int - c {.serializationKey: "_c".}: string + ```nim + type MyObj = object + a {.dontSerialize.}: int + b {.defaultDeserialize: 5.}: int + c {.serializationKey: "_c".}: string + ``` - Adopting type for gui inspector in a game engine: -.. code-block:: nim - type MyComponent = object - position {.editable, animatable.}: Vector3 - alpha {.editRange: [0.0..1.0], animatable.}: float32 + ```nim + type MyComponent = object + position {.editable, animatable.}: Vector3 + alpha {.editRange: [0.0..1.0], animatable.}: float32 + ``` Macro pragmas ------------- -All macros and templates can also be used as pragmas. They can be attached -to routines (procs, iterators, etc), type names or type expressions. The -compiler will perform the following simple syntactic transformations: +Macros and templates can sometimes be called with the pragma syntax. Cases +where this is possible include when attached to routine (procs, iterators, etc.) +declarations or routine type expressions. The compiler will perform the +following simple syntactic transformations: -.. code-block:: nim + ```nim template command(name: string, def: untyped) = discard proc p() {.command("print").} = discard + ``` This is translated to: -.. code-block:: nim + ```nim command("print"): proc p() = discard + ``` ------ -.. code-block:: nim + ```nim type AsyncEventHandler = proc (x: Event) {.async.} + ``` This is translated to: -.. code-block:: nim + ```nim type AsyncEventHandler = async(proc (x: Event)) + ``` ------ -.. code-block:: nim - type - MyObject {.schema: "schema.protobuf".} = object - -This is translated to a call to the ``schema`` macro with a `nnkTypeDef` -AST node capturing both the left-hand side and right-hand side of the -definition. The macro can return a potentially modified `nnkTypeDef` tree -which will replace the original row in the type section. - -When multiple macro pragmas are applied to the same definition, the -compiler will apply them consequently from left to right. Each macro -will receive as input the output of the previous one. +When multiple macro pragmas are applied to the same definition, the first one +from left to right will be evaluated. This macro can then choose to keep +the remaining macro pragmas in its output, and those will be evaluated in +the same way. +There are a few more applications of macro pragmas, such as in type, +variable and constant declarations, but this behavior is considered to be +experimental and is documented in the [experimental manual]( +manual_experimental.html#extended-macro-pragmas) instead. Foreign function interface @@ -7178,117 +8578,186 @@ are documented here. Importc pragma -------------- -The ``importc`` pragma provides a means to import a proc or a variable +The `importc` pragma provides a means to import a proc or a variable from C. The optional argument is a string containing the C identifier. If the argument is missing, the C name is the Nim identifier *exactly as spelled*: -.. code-block:: + ```nim proc printf(formatstr: cstring) {.header: "<stdio.h>", importc: "printf", varargs.} + ``` -When ``importc`` is applied to a ``let`` statement it can omit its value which -will then be expected to come from C. This can be used to import a C ``const``: +When `importc` is applied to a `let` statement it can omit its value which +will then be expected to come from C. This can be used to import a C `const`:c:\: -.. code-block:: + ```nim {.emit: "const int cconst = 42;".} let cconst {.importc, nodecl.}: cint assert cconst == 42 + ``` Note that this pragma has been abused in the past to also work in the -js backend for js objects and functions. : Other backends do provide +JS backend for JS objects and functions. Other backends do provide the same feature under the same name. Also, when the target language is not set to C, other pragmas are available: - * `importcpp <manual.html#implementation-specific-pragmas-importcpp-pragma>`_ - * `importobjc <manual.html#implementation-specific-pragmas-importobjc-pragma>`_ - * `importjs <manual.html#implementation-specific-pragmas-importjs-pragma>`_ + * [importcpp][importcpp pragma] + * [importobjc][importobjc pragma] + * [importjs][importjs pragma] + +The string literal passed to `importc` can be a format string: -.. code-block:: Nim + ```Nim proc p(s: cstring) {.importc: "prefix$1".} + ``` -In the example the external name of ``p`` is set to ``prefixp``. Only ``$1`` +In the example, the external name of `p` is set to `prefixp`. Only ``$1`` is available and a literal dollar sign must be written as ``$$``. Exportc pragma -------------- -The ``exportc`` pragma provides a means to export a type, a variable, or a +The `exportc` pragma provides a means to export a type, a variable, or a procedure to C. Enums and constants can't be exported. The optional argument -is a string containing the C identifier. If the argument is missing, the C +is a string containing the C identifier. If the argument is missing, the C name is the Nim identifier *exactly as spelled*: -.. code-block:: Nim + ```Nim proc callme(formatstr: cstring) {.exportc: "callMe", varargs.} + ``` Note that this pragma is somewhat of a misnomer: Other backends do provide the same feature under the same name. -The string literal passed to ``exportc`` can be a format string: +The string literal passed to `exportc` can be a format string: -.. code-block:: Nim + ```Nim proc p(s: string) {.exportc: "prefix$1".} = echo s + ``` -In the example the external name of ``p`` is set to ``prefixp``. Only ``$1`` +In the example, the external name of `p` is set to `prefixp`. Only ``$1`` is available and a literal dollar sign must be written as ``$$``. -If the symbol should also be exported to a dynamic library, the ``dynlib`` -pragma should be used in addition to the ``exportc`` pragma. See -`Dynlib pragma for export <#foreign-function-interface-dynlib-pragma-for-export>`_. +If the symbol should also be exported to a dynamic library, the `dynlib` +pragma should be used in addition to the `exportc` pragma. See +[Dynlib pragma for export]. Extern pragma ------------- -Like ``exportc`` or ``importc``, the ``extern`` pragma affects name -mangling. The string literal passed to ``extern`` can be a format string: +Like `exportc` or `importc`, the `extern` pragma affects name +mangling. The string literal passed to `extern` can be a format string: -.. code-block:: Nim + ```Nim proc p(s: string) {.extern: "prefix$1".} = echo s + ``` -In the example the external name of ``p`` is set to ``prefixp``. Only ``$1`` +In the example, the external name of `p` is set to `prefixp`. Only ``$1`` is available and a literal dollar sign must be written as ``$$``. Bycopy pragma ------------- -The ``bycopy`` pragma can be applied to an object or tuple type and -instructs the compiler to pass the type by value to procs: +The `bycopy` pragma can be applied to an object or tuple type or a proc param. It instructs the compiler to pass the type by value to procs: -.. code-block:: nim + ```nim type Vector {.bycopy.} = object x, y, z: float + ``` +The Nim compiler automatically determines whether a parameter is passed by value or +by reference based on the parameter type's size. If a parameter must be passed by value +or by reference, (such as when interfacing with a C library) use the bycopy or byref pragmas. +Notice params marked as `byref` takes precedence over types marked as `bycopy`. Byref pragma ------------ -The ``byref`` pragma can be applied to an object or tuple type and instructs -the compiler to pass the type by reference (hidden pointer) to procs. +The `byref` pragma can be applied to an object or tuple type or a proc param. +When applied to a type it instructs the compiler to pass the type by reference +(hidden pointer) to procs. When applied to a param it will take precedence, even +if the the type was marked as `bycopy`. When an `importc` type has a `byref` pragma or +parameters are marked as `byref` in an `importc` proc, these params translate to pointers. +When an `importcpp` type has a `byref` pragma, these params translate to +C++ references `&`. + ```Nim + {.emit: """/*TYPESECTION*/ + typedef struct { + int x; + } CStruct; + """.} + + {.emit: """ + #ifdef __cplusplus + extern "C" + #endif + int takesCStruct(CStruct* x) { + return x->x; + } + """.} + + type + CStruct {.importc, byref.} = object + x: cint + + proc takesCStruct(x: CStruct): cint {.importc.} + ``` + + or + + + ```Nim + type + CStruct {.importc.} = object + x: cint + + proc takesCStruct(x {.byref.}: CStruct): cint {.importc.} + ``` + + ```Nim + {.emit: """/*TYPESECTION*/ + struct CppStruct { + int x; + + int takesCppStruct(CppStruct& y) { + return x + y.x; + } + }; + """.} + + type + CppStruct {.importcpp, byref.} = object + x: cint + + proc takesCppStruct(x, y: CppStruct): cint {.importcpp.} + ``` Varargs pragma -------------- -The ``varargs`` pragma can be applied to procedures only (and procedure +The `varargs` pragma can be applied to procedures only (and procedure types). It tells Nim that the proc can take a variable number of parameters after the last specified parameter. Nim string values will be converted to C strings automatically: -.. code-block:: Nim - proc printf(formatstr: cstring) {.nodecl, varargs.} + ```Nim + proc printf(formatstr: cstring) {.header: "<stdio.h>", varargs.} printf("hallo %s", "world") # "world" will be passed as C string + ``` Union pragma ------------ -The ``union`` pragma can be applied to any ``object`` type. It means all -of the object's fields are overlaid in memory. This produces a ``union`` -instead of a ``struct`` in the generated C/C++ code. The object declaration +The `union` pragma can be applied to any `object` type. It means all +of an object's fields are overlaid in memory. This produces a `union`:c: +instead of a `struct`:c: in the generated C/C++ code. The object declaration then must not use inheritance or any GC'ed memory but this is currently not checked. @@ -7298,7 +8767,7 @@ should scan unions conservatively. Packed pragma ------------- -The ``packed`` pragma can be applied to any ``object`` type. It ensures +The `packed` pragma can be applied to any `object` type. It ensures that the fields of an object are packed back-to-back in memory. It is useful to store packets or messages from/to network or hardware drivers, and for interoperability with C. Combining packed pragma with inheritance is not @@ -7310,40 +8779,42 @@ a static error. Usage with inheritance should be defined and documented. Dynlib pragma for import ------------------------ -With the ``dynlib`` pragma a procedure or a variable can be imported from +With the `dynlib` pragma, a procedure or a variable can be imported from a dynamic library (``.dll`` files for Windows, ``lib*.so`` files for UNIX). The non-optional argument has to be the name of the dynamic library: -.. code-block:: Nim + ```Nim proc gtk_image_new(): PGtkWidget {.cdecl, dynlib: "libgtk-x11-2.0.so", importc.} + ``` In general, importing a dynamic library does not require any special linker options or linking with import libraries. This also implies that no *devel* packages need to be installed. -The ``dynlib`` import mechanism supports a versioning scheme: +The `dynlib` import mechanism supports a versioning scheme: -.. code-block:: nim + ```nim proc Tcl_Eval(interp: pTcl_Interp, script: cstring): int {.cdecl, importc, dynlib: "libtcl(|8.5|8.4|8.3).so.(1|0)".} + ``` -At runtime the dynamic library is searched for (in this order):: +At runtime, the dynamic library is searched for (in this order): - libtcl.so.1 - libtcl.so.0 - libtcl8.5.so.1 - libtcl8.5.so.0 - libtcl8.4.so.1 - libtcl8.4.so.0 - libtcl8.3.so.1 - libtcl8.3.so.0 + libtcl.so.1 + libtcl.so.0 + libtcl8.5.so.1 + libtcl8.5.so.0 + libtcl8.4.so.1 + libtcl8.4.so.0 + libtcl8.3.so.1 + libtcl8.3.so.0 -The ``dynlib`` pragma supports not only constant strings as argument but also +The `dynlib` pragma supports not only constant strings as an argument but also string expressions in general: -.. code-block:: nim - import os + ```nim + import std/os proc getDllName: string = result = "mylib.dll" @@ -7353,113 +8824,69 @@ string expressions in general: quit("could not load dynamic library") proc myImport(s: cstring) {.cdecl, importc, dynlib: getDllName().} + ``` **Note**: Patterns like ``libtcl(|8.5|8.4).so`` are only supported in constant strings, because they are precompiled. -**Note**: Passing variables to the ``dynlib`` pragma will fail at runtime +**Note**: Passing variables to the `dynlib` pragma will fail at runtime because of order of initialization problems. -**Note**: A ``dynlib`` import can be overridden with -the ``--dynlibOverride:name`` command line option. The Compiler User Guide -contains further information. +**Note**: A `dynlib` import can be overridden with +the `--dynlibOverride:name`:option: command-line option. The +[Compiler User Guide](nimc.html) contains further information. Dynlib pragma for export ------------------------ -With the ``dynlib`` pragma a procedure can also be exported to +With the `dynlib` pragma, a procedure can also be exported to a dynamic library. The pragma then has no argument and has to be used in -conjunction with the ``exportc`` pragma: +conjunction with the `exportc` pragma: -.. code-block:: Nim + ```Nim proc exportme(): int {.cdecl, exportc, dynlib.} + ``` This is only useful if the program is compiled as a dynamic library via the -``--app:lib`` command line option. - +`--app:lib`:option: command-line option. Threads ======= -To enable thread support the ``--threads:on`` command line switch needs to -be used. The ``system`` module then contains several threading primitives. -See the `threads <threads.html>`_ and `channels <channels.html>`_ modules -for the low level thread API. There are also high level parallelism constructs -available. See `spawn <manual_experimental.html#parallel-amp-spawn>`_ for +The `--threads:on`:option: command-line switch is enabled by default. The [typedthreads module](typedthreads.html) module then contains several threading primitives. See [spawn](manual_experimental.html#parallel-amp-spawn) for further details. -Nim's memory model for threads is quite different than that of other common -programming languages (C, Pascal, Java): Each thread has its own (garbage -collected) heap and sharing of memory is restricted to global variables. This -helps to prevent race conditions. GC efficiency is improved quite a lot, -because the GC never has to stop other threads and see what they reference. +The only ways to create a thread is via `spawn` or `createThread`. Thread pragma ------------- A proc that is executed as a new thread of execution should be marked by the -``thread`` pragma for reasons of readability. The compiler checks for +`thread` pragma for reasons of readability. The compiler checks for violations of the `no heap sharing restriction`:idx:\: This restriction implies that it is invalid to construct a data structure that consists of memory -allocated from different (thread local) heaps. - -A thread proc is passed to ``createThread`` or ``spawn`` and invoked -indirectly; so the ``thread`` pragma implies ``procvar``. - - -GC safety ---------- - -We call a proc ``p`` `GC safe`:idx: when it doesn't access any global variable -that contains GC'ed memory (``string``, ``seq``, ``ref`` or a closure) either -directly or indirectly through a call to a GC unsafe proc. - -The `gcsafe`:idx: annotation can be used to mark a proc to be gcsafe, -otherwise this property is inferred by the compiler. Note that ``noSideEffect`` -implies ``gcsafe``. The only way to create a thread is via ``spawn`` or -``createThread``. The invoked proc must not use ``var`` parameters nor must -any of its parameters contain a ``ref`` or ``closure`` type. This enforces -the *no heap sharing restriction*. - -Routines that are imported from C are always assumed to be ``gcsafe``. -To disable the GC-safety checking the ``--threadAnalysis:off`` command line -switch can be used. This is a temporary workaround to ease the porting effort -from old code to the new threading model. - -To override the compiler's gcsafety analysis a ``{.gcsafe.}`` pragma block can -be used: - -.. code-block:: nim - - var - someGlobal: string = "some string here" - perThread {.threadvar.}: string - - proc setPerThread() = - {.gcsafe.}: - deepCopy(perThread, someGlobal) - +allocated from different (thread-local) heaps. -See also: +A thread proc can be passed to `createThread` or `spawn`. -- `Shared heap memory management. <gc.html>`_. Threadvar pragma ---------------- -A variable can be marked with the ``threadvar`` pragma, which makes it a +A variable can be marked with the `threadvar` pragma, which makes it a `thread-local`:idx: variable; Additionally, this implies all the effects -of the ``global`` pragma. +of the `global` pragma. -.. code-block:: nim + ```nim var checkpoints* {.threadvar.}: seq[string] + ``` -Due to implementation restrictions thread local variables cannot be -initialized within the ``var`` section. (Every thread local variable needs to +Due to implementation restrictions, thread-local variables cannot be +initialized within the `var` section. (Every thread-local variable needs to be replicated at thread creation.) @@ -7468,4 +8895,139 @@ Threads and exceptions The interaction between threads and exceptions is simple: A *handled* exception in one thread cannot affect any other thread. However, an *unhandled* exception -in one thread terminates the whole *process*! +in one thread terminates the whole *process*. + + +Guards and locks +================ + +Nim provides common low level concurrency mechanisms like locks, atomic +intrinsics or condition variables. + +Nim significantly improves on the safety of these features via additional +pragmas: + +1) A `guard`:idx: annotation is introduced to prevent data races. +2) Every access of a guarded memory location needs to happen in an + appropriate `locks`:idx: statement. + + +Guards and locks sections +------------------------- + +### Protecting global variables + +Object fields and global variables can be annotated via a `guard` pragma: + + ```nim + import std/locks + + var glock: Lock + var gdata {.guard: glock.}: int + ``` + +The compiler then ensures that every access of `gdata` is within a `locks` +section: + + ```nim + proc invalid = + # invalid: unguarded access: + echo gdata + + proc valid = + # valid access: + {.locks: [glock].}: + echo gdata + ``` + +Top level accesses to `gdata` are always allowed so that it can be initialized +conveniently. It is *assumed* (but not enforced) that every top level statement +is executed before any concurrent action happens. + +The `locks` section deliberately looks ugly because it has no runtime +semantics and should not be used directly! It should only be used in templates +that also implement some form of locking at runtime: + + ```nim + template lock(a: Lock; body: untyped) = + pthread_mutex_lock(a) + {.locks: [a].}: + try: + body + finally: + pthread_mutex_unlock(a) + ``` + + +The guard does not need to be of any particular type. It is flexible enough to +model low level lockfree mechanisms: + + ```nim + var dummyLock {.compileTime.}: int + var atomicCounter {.guard: dummyLock.}: int + + template atomicRead(x): untyped = + {.locks: [dummyLock].}: + memoryReadBarrier() + x + + echo atomicRead(atomicCounter) + ``` + + +The `locks` pragma takes a list of lock expressions `locks: [a, b, ...]` +in order to support *multi lock* statements. + + +### Protecting general locations + +The `guard` annotation can also be used to protect fields within an object. +The guard then needs to be another field within the same object or a +global variable. + +Since objects can reside on the heap or on the stack, this greatly enhances +the expressiveness of the language: + + ```nim + import std/locks + + type + ProtectedCounter = object + v {.guard: L.}: int + L: Lock + + proc incCounters(counters: var openArray[ProtectedCounter]) = + for i in 0..counters.high: + lock counters[i].L: + inc counters[i].v + ``` + +The access to field `x.v` is allowed since its guard `x.L` is active. +After template expansion, this amounts to: + + ```nim + proc incCounters(counters: var openArray[ProtectedCounter]) = + for i in 0..counters.high: + pthread_mutex_lock(counters[i].L) + {.locks: [counters[i].L].}: + try: + inc counters[i].v + finally: + pthread_mutex_unlock(counters[i].L) + ``` + +There is an analysis that checks that `counters[i].L` is the lock that +corresponds to the protected location `counters[i].v`. This analysis is called +`path analysis`:idx: because it deals with paths to locations +like `obj.field[i].fieldB[j]`. + +The path analysis is **currently unsound**, but that doesn't make it useless. +Two paths are considered equivalent if they are syntactically the same. + +This means the following compiles (for now) even though it really should not: + + ```nim + {.locks: [a[i].L].}: + inc i + access a[i].v + ``` diff --git a/doc/manual/var_t_return.rst b/doc/manual/var_t_return.md index 2235d5aa4..15d908c74 100644 --- a/doc/manual/var_t_return.rst +++ b/doc/manual/var_t_return.md @@ -1,9 +1,12 @@ -Memory safety for returning by ``var T`` is ensured by a simple borrowing -rule: If ``result`` does not refer to a location pointing to the heap -(that is in ``result = X`` the ``X`` involves a ``ptr`` or ``ref`` access) +.. default-role:: code +.. include:: ../rstcommon.rst + +Memory safety for returning by `var T` is ensured by a simple borrowing +rule: If `result` does not refer to a location pointing to the heap +(that is in `result = X` the `X` involves a `ptr` or `ref` access) then it has to be derived from the routine's first parameter: -.. code-block:: nim + ```nim proc forward[T](x: var T): var T = result = x # ok, derived from the first parameter. @@ -11,10 +14,11 @@ then it has to be derived from the routine's first parameter: var x: int # we know 'forward' provides a view into the location derived from # its first argument 'x'. - result = forward(x) # Error: location is derived from ``x`` + result = forward(x) # Error: location is derived from `x` # which is not p's first parameter and lives # on the stack. + ``` -In other words, the lifetime of what ``result`` points to is attached to the +In other words, the lifetime of what `result` points to is attached to the lifetime of the first parameter and that is enough knowledge to verify -memory safety at the callsite. +memory safety at the call site. diff --git a/doc/manual_experimental.md b/doc/manual_experimental.md new file mode 100644 index 000000000..da51d59ad --- /dev/null +++ b/doc/manual_experimental.md @@ -0,0 +1,2669 @@ +========================= +Nim Experimental Features +========================= + +:Authors: Andreas Rumpf +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + + +About this document +=================== + +This document describes features of Nim that are to be considered experimental. +Some of these are not covered by the `.experimental` pragma or +`--experimental`:option: switch because they are already behind a special syntax and +one may want to use Nim libraries using these features without using them +oneself. + +.. note:: Unless otherwise indicated, these features are not to be removed, + but refined and overhauled. + + +Void type +========= + +The `void` type denotes the absence of any value, i.e. it is the type that contains no values. Consequently, no value can be provided for parameters of +type `void`, and no value can be returned from a function with return type `void`: + + ```nim + proc nothing(x, y: void): void = + echo "ha" + + nothing() # writes "ha" to stdout + ``` + +The `void` type is particularly useful for generic code: + + ```nim + proc callProc[T](p: proc (x: T), x: T) = + when T is void: + p() + else: + p(x) + + proc intProc(x: int) = discard + proc emptyProc() = discard + + callProc[int](intProc, 12) + callProc[void](emptyProc) + ``` + +However, a `void` type cannot be inferred in generic code: + + ```nim + callProc(emptyProc) + # Error: type mismatch: got (proc ()) + # but expected one of: + # callProc(p: proc (T), x: T) + ``` + +The `void` type is only valid for parameters and return types; other symbols +cannot have the type `void`. + +Generic `define` pragma +======================= + +Aside the [typed define pragmas for constants](manual.html#implementation-specific-pragmas-compileminustime-define-pragmas), +there is a generic `{.define.}` pragma that interprets the value of the define +based on the type of the constant value. + + ```nim + const foo {.define: "package.foo".} = 123 + const bar {.define: "package.bar".} = false + ``` + + ```cmd + nim c -d:package.foo=456 -d:package.bar foobar.nim + ``` + +The following types are supported: + +* `string` and `cstring` +* Signed and unsigned integer types +* `bool` +* Enums + +Top-down type inference +======================= + +In expressions such as: + +```nim +let a: T = ex +``` + +Normally, the compiler type checks the expression `ex` by itself, then +attempts to statically convert the type-checked expression to the given type +`T` as much as it can, while making sure it matches the type. The extent of +this process is limited however due to the expression usually having +an assumed type that might clash with the given type. + +With top-down type inference, the expression is type checked with the +extra knowledge that it is supposed to be of type `T`. For example, +the following code is does not compile with the former method, but +compiles with top-down type inference: + +```nim +let foo: (float, uint8, cstring) = (1, 2, "abc") +``` + +The tuple expression has an expected type of `(float, uint8, cstring)`. +Since it is a tuple literal, we can use this information to assume the types +of its elements. The expected types for the expressions `1`, `2` and `"abc"` +are respectively `float`, `uint8`, and `cstring`; and these expressions can be +statically converted to these types. + +Without this information, the type of the tuple expression would have been +assumed to be `(int, int, string)`. Thus the type of the tuple expression +would not match the type of the variable, and an error would be given. + +The extent of this varies, but there are some notable special cases. + + +Inferred generic parameters +--------------------------- + +In expressions making use of generic procs or templates, the expected +(unbound) types are often able to be inferred based on context. +This feature has to be enabled via `{.experimental: "inferGenericTypes".}` + + ```nim test = "nim c $1" + {.experimental: "inferGenericTypes".} + + import std/options + + var x = newSeq[int](1) + # Do some work on 'x'... + + # Works! + # 'x' is 'seq[int]' so 'newSeq[int]' is implied + x = newSeq(10) + + # Works! + # 'T' of 'none' is bound to the 'T' of 'noneProducer', passing it along. + # Effectively 'none.T = noneProducer.T' + proc noneProducer[T](): Option[T] = none() + let myNone = noneProducer[int]() + + # Also works + # 'myOtherNone' binds its 'T' to 'float' and 'noneProducer' inherits it + # noneProducer.T = myOtherNone.T + let myOtherNone: Option[float] = noneProducer() + + # Works as well + # none.T = myOtherOtherNone.T + let myOtherOtherNone: Option[int] = none() + ``` + +This is achieved by reducing the types on the lhs and rhs until the *lhs* is left with only types such as `T`. +While lhs and rhs are reduced together, this does *not* mean that the *rhs* will also only be left +with a flat type `Z`, it may be of the form `MyType[Z]`. + +After the types have been reduced, the types `T` are bound to the types that are left on the rhs. + +If bindings *cannot be inferred*, compilation will fail and manual specification is required. + +An example for *failing inference* can be found when passing a generic expression +to a function/template call: + + ```nim test = "nim c $1" status = 1 + {.experimental: "inferGenericTypes".} + + proc myProc[T](a, b: T) = discard + + # Fails! Unable to infer that 'T' is supposed to be 'int' + myProc(newSeq[int](), newSeq(1)) + + # Works! Manual specification of 'T' as 'int' necessary + myProc(newSeq[int](), newSeq[int](1)) + ``` + +Combination of generic inference with the `auto` type is also unsupported: + + ```nim test = "nim c $1" status = 1 + {.experimental: "inferGenericTypes".} + + proc produceValue[T]: auto = default(T) + let a: int = produceValue() # 'auto' cannot be inferred here + ``` + +**Note**: The described inference does not permit the creation of overrides based on +the return type of a procedure. It is a mapping mechanism that does not attempt to +perform deeper inference, nor does it modify what is a valid override. + + ```nim test = "nim c $1" status = 1 + # Doesn't affect the following code, it is invalid either way + {.experimental: "inferGenericTypes".} + + proc a: int = 0 + proc a: float = 1.0 # Fails! Invalid code and not recommended + ``` + + +Sequence literals +----------------- + +Top-down type inference applies to sequence literals. + +```nim +let x: seq[seq[float]] = @[@[1, 2, 3], @[4, 5, 6]] +``` + +This behavior is tied to the `@` overloads in the `system` module, +so overloading `@` can disable this behavior. This can be circumvented by +specifying the `` system.`@` `` overload. + +```nim +proc `@`(x: string): string = "@" & x + +# does not compile: +let x: seq[float] = @[1, 2, 3] +# compiles: +let x: seq[float] = system.`@`([1, 2, 3]) +``` + + +Package level objects +===================== + +Every Nim module resides in a (nimble) package. An object type can be attached +to the package it resides in. If that is done, the type can be referenced from +other modules as an `incomplete`:idx: object type. This feature allows to +break up recursive type dependencies across module boundaries. Incomplete +object types are always passed `byref` and can only be used in pointer like +contexts (`var/ref/ptr IncompleteObject`) in general, since the compiler does +not yet know the size of the object. To complete an incomplete object, +the `package` pragma has to be used. `package` implies `byref`. + +As long as a type `T` is incomplete, no runtime type information for `T` is +available. + + +Example: + + ```nim + # module A (in an arbitrary package) + type + Pack.SomeObject = object # declare as incomplete object of package 'Pack' + Triple = object + a, b, c: ref SomeObject # pointers to incomplete objects are allowed + + # Incomplete objects can be used as parameters: + proc myproc(x: SomeObject) = discard + ``` + + + ```nim + # module B (in package "Pack") + type + SomeObject* {.package.} = object # Use 'package' to complete the object + s, t: string + x, y: int + ``` + +This feature will likely be superseded in the future by support for +recursive module dependencies. + + +Importing private symbols +========================= + +In some situations, it may be useful to import all symbols (public or private) +from a module. The syntax `import foo {.all.}` can be used to import all +symbols from the module `foo`. Note that importing private symbols is +generally not recommended. + +See also the experimental [importutils](importutils.html) module. + + +Code reordering +=============== + +The code reordering feature can implicitly rearrange procedure, template, and +macro definitions along with variable declarations and initializations at the top +level scope so that, to a large extent, a programmer should not have to worry +about ordering definitions correctly or be forced to use forward declarations to +preface definitions inside a module. + +.. + NOTE: The following was documentation for the code reordering precursor, + which was {.noForward.}. + + In this mode, procedure definitions may appear out of order and the compiler + will postpone their semantic analysis and compilation until it actually needs + to generate code using the definitions. In this regard, this mode is similar + to the modus operandi of dynamic scripting languages, where the function + calls are not resolved until the code is executed. Here is the detailed + algorithm taken by the compiler: + + 1. When a callable symbol is first encountered, the compiler will only note + the symbol callable name and it will add it to the appropriate overload set + in the current scope. At this step, it won't try to resolve any of the type + expressions used in the signature of the symbol (so they can refer to other + not yet defined symbols). + + 2. When a top level call is encountered (usually at the very end of the + module), the compiler will try to determine the actual types of all of the + symbols in the matching overload set. This is a potentially recursive process + as the signatures of the symbols may include other call expressions, whose + types will be resolved at this point too. + + 3. Finally, after the best overload is picked, the compiler will start + compiling the body of the respective symbol. This in turn will lead the + compiler to discover more call expressions that need to be resolved and steps + 2 and 3 will be repeated as necessary. + + Please note that if a callable symbol is never used in this scenario, its + body will never be compiled. This is the default behavior leading to best + compilation times, but if exhaustive compilation of all definitions is + required, using `nim check` provides this option as well. + +Example: + + ```nim + {.experimental: "codeReordering".} + + proc foo(x: int) = + bar(x) + + proc bar(x: int) = + echo(x) + + foo(10) + ``` + +Variables can also be reordered as well. Variables that are *initialized* (i.e. +variables that have their declaration and assignment combined in a single +statement) can have their entire initialization statement reordered. Be wary of +what code is executed at the top level: + + ```nim + {.experimental: "codeReordering".} + + proc a() = + echo(foo) + + var foo = 5 + + a() # outputs: "5" + ``` + +.. + TODO: Let's table this for now. This is an *experimental feature* and so the + specific manner in which `declared` operates with it can be decided in + eventuality, because right now it works a bit weirdly. + + The values of expressions involving `declared` are decided *before* the + code reordering process, and not after. As an example, the output of this + code is the same as it would be with code reordering disabled. + + ```nim + {.experimental: "codeReordering".} + + proc x() = + echo(declared(foo)) + + var foo = 4 + + x() # "false" + ``` + +It is important to note that reordering *only* works for symbols at top level +scope. Therefore, the following will *fail to compile:* + + ```nim + {.experimental: "codeReordering".} + + proc a() = + b() + proc b() = + echo("Hello!") + + a() + ``` + +This feature will likely be replaced with a better solution to remove +the need for forward declarations. + +Special Operators +================= + +dot operators +------------- + +.. note:: Dot operators are still experimental and so need to be enabled + via `{.experimental: "dotOperators".}`. + +Nim offers a special family of dot operators that can be used to +intercept and rewrite proc call and field access attempts, referring +to previously undeclared symbol names. They can be used to provide a +fluent interface to objects lying outside the static confines of the +type system such as values from dynamic scripting languages +or dynamic file formats such as JSON or XML. + +When Nim encounters an expression that cannot be resolved by the +standard overload resolution rules, the current scope will be searched +for a dot operator that can be matched against a re-written form of +the expression, where the unknown field or proc name is passed to +an `untyped` parameter: + + ```nim + a.b # becomes `.`(a, b) + a.b(c, d) # becomes `.`(a, b, c, d) + ``` + +The matched dot operators can be symbols of any callable kind (procs, +templates and macros), depending on the desired effect: + + ```nim + template `.`(js: PJsonNode, field: untyped): JSON = js[astToStr(field)] + + var js = parseJson("{ x: 1, y: 2}") + echo js.x # outputs 1 + echo js.y # outputs 2 + ``` + +The following dot operators are available: + +operator `.` +------------ +This operator will be matched against both field accesses and method calls. + +operator `.()` +--------------- +This operator will be matched exclusively against method calls. It has higher +precedence than the `.` operator and this allows one to handle expressions like +`x.y` and `x.y()` differently if one is interfacing with a scripting language +for example. + +operator `.=` +------------- +This operator will be matched against assignments to missing fields. + + ```nim + a.b = c # becomes `.=`(a, b, c) + ``` + +Call operator +------------- +The call operator, `()`, matches all kinds of unresolved calls and takes +precedence over dot operators, however it does not match missing overloads +for existing routines. The experimental `callOperator` switch must be enabled +to use this operator. + + ```nim + {.experimental: "callOperator".} + + template `()`(a: int, b: float): untyped = $(a, b) + + block: + let a = 1.0 + let b = 2 + doAssert b(a) == `()`(b, a) + doAssert a.b == `()`(b, a) + + block: + let a = 1.0 + proc b(): int = 2 + doAssert not compiles(b(a)) + doAssert not compiles(a.b) # `()` not called + + block: + let a = 1.0 + proc b(x: float): int = int(x + 1) + let c = 3.0 + + doAssert not compiles(a.b(c)) # gives a type mismatch error same as b(a, c) + doAssert (a.b)(c) == `()`(a.b, c) + ``` + + +Extended macro pragmas +====================== + +Macro pragmas as described in [the manual](manual.html#userminusdefined-pragmas-macro-pragmas) +can also be applied to type, variable and constant declarations. + +For types: + + ```nim + type + MyObject {.schema: "schema.protobuf".} = object + ``` + +This is translated to a call to the `schema` macro with a `nnkTypeDef` +AST node capturing the left-hand side, remaining pragmas and the right-hand +side of the definition. The macro can return either a type section or +another `nnkTypeDef` node, both of which will replace the original row +in the type section. + +In the future, this `nnkTypeDef` argument may be replaced with a unary +type section node containing the type definition, or some other node that may +be more convenient to work with. The ability to return nodes other than type +definitions may also be supported, however currently this is not convenient +when dealing with mutual type recursion. For now, macros can return an unused +type definition where the right-hand node is of kind `nnkStmtListType`. +Declarations in this node will be attached to the same scope as +the parent scope of the type section. + +------ + +For variables and constants, it is largely the same, except a unary node with +the same kind as the section containing a single definition is passed to macros, +and macros can return any expression. + + ```nim + var + a = ... + b {.importc, foo, nodecl.} = ... + c = ... + ``` + +Assuming `foo` is a macro or a template, this is roughly equivalent to: + + ```nim + var a = ... + foo: + var b {.importc, nodecl.} = ... + var c = ... + ``` + + +Symbols as template/macro calls (alias syntax) +============================================== + +Templates and macros that have no generic parameters and no required arguments +can be called as lone symbols, i.e. without parentheses. This is useful for +repeated uses of complex expressions that cannot conveniently be represented +as runtime values. + + ```nim + type Foo = object + bar: int + + var foo = Foo(bar: 10) + template bar: int = foo.bar + assert bar == 10 + bar = 15 + assert bar == 15 + ``` + + +Not nil annotation +================== + +**Note:** This is an experimental feature. It can be enabled with +`{.experimental: "notnil".}`. + +All types for which `nil` is a valid value can be annotated with the +`not nil` annotation to exclude `nil` as a valid value. Note that only local +symbols are checked. + + ```nim + {.experimental: "notnil".} + + type + TObj = object + PObject = ref TObj not nil + TProc = (proc (x, y: int)) not nil + + proc p(x: PObject) = + echo "not nil" + + # compiler catches this: + p(nil) + + # and also this: + proc foo = + var x: PObject + p(x) + + foo() + ``` + +The compiler ensures that every code path initializes variables which contain +non-nilable pointers. The details of this analysis are still to be specified +here. + +.. include:: manual_experimental_strictnotnil.md + + +Aliasing restrictions in parameter passing +========================================== + +.. note:: The aliasing restrictions are currently not enforced by the + implementation and need to be fleshed out further. + +"Aliasing" here means that the underlying storage locations overlap in memory +at runtime. An "output parameter" is a parameter of type `var T`, +an input parameter is any parameter that is not of type `var`. + +1. Two output parameters should never be aliased. +2. An input and an output parameter should not be aliased. +3. An output parameter should never be aliased with a global or thread local + variable referenced by the called proc. +4. An input parameter should not be aliased with a global or thread local + variable updated by the called proc. + +One problem with rules 3 and 4 is that they affect specific global or thread +local variables, but Nim's effect tracking only tracks "uses no global variable" +via `.noSideEffect`. The rules 3 and 4 can also be approximated by a different rule: + +5. A global or thread local variable (or a location derived from such a location) + can only passed to a parameter of a `.noSideEffect` proc. + + +Strict funcs +============ + +Since version 1.4, a stricter definition of "side effect" is available. +In addition to the existing rule that a side effect is calling a function +with side effects, the following rule is also enforced: + +A store to the heap via a `ref` or `ptr` indirection is not allowed. + +For example: + + ```nim + {.experimental: "strictFuncs".} + + type + Node = ref object + le, ri: Node + data: string + + func len(n: Node): int = + # valid: len does not have side effects + var it = n + while it != nil: + inc result + it = it.ri + + func mut(n: Node) = + var it = n + while it != nil: + it.data = "yeah" # forbidden mutation + it = it.ri + + ``` + + +View types +========== + +.. tip:: `--experimental:views`:option: is more effective + with `--experimental:strictFuncs`:option:. + +A view type is a type that is or contains one of the following types: + +- `lent T` (view into `T`) +- `openArray[T]` (pair of (pointer to array of `T`, size)) + +For example: + + ```nim + type + View1 = openArray[byte] + View2 = lent string + View3 = Table[openArray[char], int] + ``` + + +Exceptions to this rule are types constructed via `ptr` or `proc`. +For example, the following types are **not** view types: + + ```nim + type + NotView1 = proc (x: openArray[int]) + NotView2 = ptr openArray[char] + NotView3 = ptr array[4, lent int] + ``` + + +The mutability aspect of a view type is not part of the type but part +of the locations it's derived from. More on this later. + +A *view* is a symbol (a let, var, const, etc.) that has a view type. + +Since version 1.4, Nim allows view types to be used as local variables. +This feature needs to be enabled via `{.experimental: "views".}`. + +A local variable of a view type *borrows* from the locations and +it is statically enforced that the view does not outlive the location +it was borrowed from. + +For example: + + ```nim + {.experimental: "views".} + + proc take(a: openArray[int]) = + echo a.len + + proc main(s: seq[int]) = + var x: openArray[int] = s # 'x' is a view into 's' + # it is checked that 'x' does not outlive 's' and + # that 's' is not mutated. + for i in 0 .. high(x): + echo x[i] + take(x) + + take(x.toOpenArray(0, 1)) # slicing remains possible + let y = x # create a view from a view + take y + # it is checked that 'y' does not outlive 'x' and + # that 'x' is not mutated as long as 'y' lives. + + + main(@[11, 22, 33]) + ``` + + +A local variable of a view type can borrow from a location +derived from a parameter, another local variable, a global `const` or `let` +symbol or a thread-local `var` or `let`. + +Let `p` the proc that is analysed for the correctness of the borrow operation. + +Let `source` be one of: + +- A formal parameter of `p`. Note that this does not cover parameters of + inner procs. +- The `result` symbol of `p`. +- A local `var` or `let` or `const` of `p`. Note that this does + not cover locals of inner procs. +- A thread-local `var` or `let`. +- A global `let` or `const`. +- A constant array/seq/object/tuple constructor. + + +Path expressions +---------------- + +A location derived from `source` is then defined as a path expression that +has `source` as the owner. A path expression `e` is defined recursively: + +- `source` itself is a path expression. +- Container access like `e[i]` is a path expression. +- Tuple access `e[0]` is a path expression. +- Object field access `e.field` is a path expression. +- `system.toOpenArray(e, ...)` is a path expression. +- Pointer dereference `e[]` is a path expression. +- An address `addr e` is a path expression. +- A type conversion `T(e)` is a path expression. +- A cast expression `cast[T](e)` is a path expression. +- `f(e, ...)` is a path expression if `f`'s return type is a view type. + Because the view can only have been borrowed from `e`, we then know + that the owner of `f(e, ...)` is `e`. + + +If a view type is used as a return type, the location must borrow from a location +that is derived from the first parameter that is passed to the proc. +See [the manual](manual.html#procedures-var-return-type) +for details about how this is done for `var T`. + +A mutable view can borrow from a mutable location, an immutable view can borrow +from both a mutable or an immutable location. + +If a view borrows from a mutable location, the view can be used to update the +location. Otherwise it cannot be used for mutations. + +The *duration* of a borrow is the span of commands beginning from the assignment +to the view and ending with the last usage of the view. + +For the duration of the borrow operation, no mutations to the borrowed locations +may be performed except via the view that borrowed from the +location. The borrowed location is said to be *sealed* during the borrow. + + ```nim + {.experimental: "views".} + + type + Obj = object + field: string + + proc dangerous(s: var seq[Obj]) = + let v: lent Obj = s[0] # seal 's' + s.setLen 0 # prevented at compile-time because 's' is sealed. + echo v.field + ``` + + +The scope of the view does not matter: + + ```nim + proc valid(s: var seq[Obj]) = + let v: lent Obj = s[0] # begin of borrow + echo v.field # end of borrow + s.setLen 0 # valid because 'v' isn't used afterwards + ``` + + +The analysis requires as much precision about mutations as is reasonably obtainable, +so it is more effective with the experimental [strict funcs] +feature. In other words `--experimental:views`:option: works better +with `--experimental:strictFuncs`:option:. + +The analysis is currently control flow insensitive: + + ```nim + proc invalid(s: var seq[Obj]) = + let v: lent Obj = s[0] + if false: + s.setLen 0 + echo v.field + ``` + +In this example, the compiler assumes that `s.setLen 0` invalidates the +borrow operation of `v` even though a human being can easily see that it +will never do that at runtime. + + +Start of a borrow +----------------- + +A borrow starts with one of the following: + +- The assignment of a non-view-type to a view-type. +- The assignment of a location that is derived from a local parameter + to a view-type. + + +End of a borrow +--------------- + +A borrow operation ends with the last usage of the view variable. + + +Reborrows +--------- + +A view `v` can borrow from multiple different locations. However, the borrow +is always the full span of `v`'s lifetime and every location that is borrowed +from is sealed during `v`'s lifetime. + + +Algorithm +--------- + +The following section is an outline of the algorithm that the current implementation +uses. The algorithm performs two traversals over the AST of the procedure or global +section of code that uses a view variable. No fixpoint iterations are performed, the +complexity of the analysis is O(N) where N is the number of nodes of the AST. + +The first pass over the AST computes the lifetime of each local variable based on +a notion of an "abstract time", in the implementation it's a simple integer that is +incremented for every visited node. + +In the second pass, information about the underlying object "graphs" is computed. +Let `v` be a parameter or a local variable. Let `G(v)` be the graph +that `v` belongs to. A graph is defined by the set of variables that belong +to the graph. Initially for all `v`: `G(v) = {v}`. Every variable can only +be part of a single graph. + +Assignments like `a = b` "connect" two variables, both variables end up in the +same graph `{a, b} = G(a) = G(b)`. Unfortunately, the pattern to look for is +much more complex than that and can involve multiple assignment targets +and sources: + + f(x, y) = g(a, b) + +connects `x` and `y` to `a` and `b`: `G(x) = G(y) = G(a) = G(b) = {x, y, a, b}`. +A type based alias analysis rules out some of these combinations, for example +a `string` value cannot possibly be connected to a `seq[int]`. + +A pattern like `v[] = value` or `v.field = value` marks `G(v)` as mutated. +After the second pass a set of disjoint graphs was computed. + +For strict functions it is then enforced that there is no graph that is both mutated +and has an element that is an immutable parameter (that is a parameter that is not +of type `var T`). + +For borrow checking, a different set of checks is performed. Let `v` be the view +and `b` the location that is borrowed from. + +- The lifetime of `v` must not exceed `b`'s lifetime. Note: The lifetime of + a parameter is the complete proc body. +- If `v` is used for a mutation, `b` must be a mutable location too. +- During `v`'s lifetime, `G(b)` can only be modified by `v` (and only if + `v` is a mutable view). +- If `v` is `result` then `b` has to be a location derived from the first + formal parameter or from a constant location. +- A view cannot be used for a read or a write access before it was assigned to. + + +Concepts +======== + +Concepts, also known as "user-defined type classes", are used to specify an +arbitrary set of requirements that the matched type must satisfy. + +Concepts are written in the following form: + + ```nim + type + Comparable = concept x, y + (x < y) is bool + + Stack[T] = concept s, var v + s.pop() is T + v.push(T) + + s.len is Ordinal + + for value in s: + value is T + ``` + +The concept matches if: + +a) all expressions within the body can be compiled for the tested type +b) all statically evaluable boolean expressions in the body are true +c) all type modifiers specified match their respective definitions + +The identifiers following the `concept` keyword represent instances of the +currently matched type. You can apply any of the standard type modifiers such +as `var`, `ref`, `ptr` and `static` to denote a more specific type of +instance. You can also apply the `type` modifier to create a named instance of +the type itself: + + ```nim + type + MyConcept = concept x, var v, ref r, ptr p, static s, type T + ... + ``` + +Within the concept body, types can appear in positions where ordinary values +and parameters are expected. This provides a more convenient way to check for +the presence of callable symbols with specific signatures: + + ```nim + type + OutputStream = concept var s + s.write(string) + ``` + +In order to check for symbols accepting `type` params, you must prefix +the type with the explicit `type` modifier. The named instance of the +type, following the `concept` keyword is also considered to have the +explicit modifier and will be matched only as a type. + + ```nim + type + # Let's imagine a user-defined casting framework with operators + # such as `val.to(string)` and `val.to(JSonValue)`. We can test + # for these with the following concept: + MyCastables = concept x + x.to(type string) + x.to(type JSonValue) + + # Let's define a couple of concepts, known from Algebra: + AdditiveMonoid* = concept x, y, type T + x + y is T + T.zero is T # require a proc such as `int.zero` or 'Position.zero' + + AdditiveGroup* = concept x, y, type T + x is AdditiveMonoid + -x is T + x - y is T + ``` + +Please note that the `is` operator allows one to easily verify the precise +type signatures of the required operations, but since type inference and +default parameters are still applied in the concept body, it's also possible +to describe usage protocols that do not reveal implementation details. + +Much like generics, concepts are instantiated exactly once for each tested type +and any static code included within the body is executed only once. + + +Concept diagnostics +------------------- + +By default, the compiler will report the matching errors in concepts only when +no other overload can be selected and a normal compilation error is produced. +When you need to understand why the compiler is not matching a particular +concept and, as a result, a wrong overload is selected, you can apply the +`explain` pragma to either the concept body or a particular call-site. + + ```nim + type + MyConcept {.explain.} = concept ... + + overloadedProc(x, y, z) {.explain.} + ``` + +This will provide Hints in the compiler output either every time the concept is +not matched or only on the particular call-site. + + +Generic concepts and type binding rules +--------------------------------------- + +The concept types can be parametric just like the regular generic types: + + ```nim + ### matrixalgo.nim + + import std/typetraits + + type + AnyMatrix*[R, C: static int; T] = concept m, var mvar, type M + M.ValueType is T + M.Rows == R + M.Cols == C + + m[int, int] is T + mvar[int, int] = T + + type TransposedType = stripGenericParams(M)[C, R, T] + + AnySquareMatrix*[N: static int, T] = AnyMatrix[N, N, T] + + AnyTransform3D* = AnyMatrix[4, 4, float] + + proc transposed*(m: AnyMatrix): m.TransposedType = + for r in 0 ..< m.R: + for c in 0 ..< m.C: + result[r, c] = m[c, r] + + proc determinant*(m: AnySquareMatrix): int = + ... + + proc setPerspectiveProjection*(m: AnyTransform3D) = + ... + + -------------- + ### matrix.nim + + type + Matrix*[M, N: static int; T] = object + data: array[M*N, T] + + proc `[]`*(M: Matrix; m, n: int): M.T = + M.data[m * M.N + n] + + proc `[]=`*(M: var Matrix; m, n: int; v: M.T) = + M.data[m * M.N + n] = v + + # Adapt the Matrix type to the concept's requirements + template Rows*(M: typedesc[Matrix]): int = M.M + template Cols*(M: typedesc[Matrix]): int = M.N + template ValueType*(M: typedesc[Matrix]): typedesc = M.T + + ------------- + ### usage.nim + + import matrix, matrixalgo + + var + m: Matrix[3, 3, int] + projectionMatrix: Matrix[4, 4, float] + + echo m.transposed.determinant + setPerspectiveProjection projectionMatrix + ``` + +When the concept type is matched against a concrete type, the unbound type +parameters are inferred from the body of the concept in a way that closely +resembles the way generic parameters of callable symbols are inferred on +call sites. + +Unbound types can appear both as params to calls such as `s.push(T)` and +on the right-hand side of the `is` operator in cases such as `x.pop is T` +and `x.data is seq[T]`. + +Unbound static params will be inferred from expressions involving the `==` +operator and also when types dependent on them are being matched: + + ```nim + type + MatrixReducer[M, N: static int; T] = concept x + x.reduce(SquareMatrix[N, T]) is array[M, int] + ``` + +The Nim compiler includes a simple linear equation solver, allowing it to +infer static params in some situations where integer arithmetic is involved. + +Just like in regular type classes, Nim discriminates between `bind once` +and `bind many` types when matching the concept. You can add the `distinct` +modifier to any of the otherwise inferable types to get a type that will be +matched without permanently inferring it. This may be useful when you need +to match several procs accepting the same wide class of types: + + ```nim + type + Enumerable[T] = concept e + for v in e: + v is T + + type + MyConcept = concept o + # this could be inferred to a type such as Enumerable[int] + o.foo is distinct Enumerable + + # this could be inferred to a different type such as Enumerable[float] + o.bar is distinct Enumerable + + # it's also possible to give an alias name to a `bind many` type class + type Enum = distinct Enumerable + o.baz is Enum + ``` + +On the other hand, using `bind once` types allows you to test for equivalent +types used in multiple signatures, without actually requiring any concrete +types, thus allowing you to encode implementation-defined types: + + ```nim + type + MyConcept = concept x + type T1 = auto + x.foo(T1) + x.bar(T1) # both procs must accept the same type + + type T2 = seq[SomeNumber] + x.alpha(T2) + x.omega(T2) # both procs must accept the same type + # and it must be a numeric sequence + ``` + +As seen in the previous examples, you can refer to generic concepts such as +`Enumerable[T]` just by their short name. Much like the regular generic types, +the concept will be automatically instantiated with the bind once auto type +in the place of each missing generic param. + +Please note that generic concepts such as `Enumerable[T]` can be matched +against concrete types such as `string`. Nim doesn't require the concept +type to have the same number of parameters as the type being matched. +If you wish to express a requirement towards the generic parameters of +the matched type, you can use a type mapping operator such as `genericHead` +or `stripGenericParams` within the body of the concept to obtain the +uninstantiated version of the type, which you can then try to instantiate +in any required way. For example, here is how one might define the classic +`Functor` concept from Haskell and then demonstrate that Nim's `Option[T]` +type is an instance of it: + + ```nim test = "nim c $1" + import std/[sugar, typetraits] + + type + Functor[A] = concept f + type MatchedGenericType = genericHead(typeof(f)) + # `f` will be a value of a type such as `Option[T]` + # `MatchedGenericType` will become the `Option` type + + f.val is A + # The Functor should provide a way to obtain + # a value stored inside it + + type T = auto + map(f, A -> T) is MatchedGenericType[T] + # And it should provide a way to map one instance of + # the Functor to a instance of a different type, given + # a suitable `map` operation for the enclosed values + + import std/options + echo Option[int] is Functor # prints true + ``` + + +Concept derived values +---------------------- + +All top level constants or types appearing within the concept body are +accessible through the dot operator in procs where the concept was successfully +matched to a concrete type: + + ```nim + type + DateTime = concept t1, t2, type T + const Min = T.MinDate + T.Now is T + + t1 < t2 is bool + + type TimeSpan = typeof(t1 - t2) + TimeSpan * int is TimeSpan + TimeSpan + TimeSpan is TimeSpan + + t1 + TimeSpan is T + + proc eventsJitter(events: Enumerable[DateTime]): float = + var + # this variable will have the inferred TimeSpan type for + # the concrete Date-like value the proc was called with: + averageInterval: DateTime.TimeSpan + + deviation: float + ... + ``` + + +Concept refinement +------------------ + +When the matched type within a concept is directly tested against a different +concept, we say that the outer concept is a refinement of the inner concept and +thus it is more-specific. When both concepts are matched in a call during +overload resolution, Nim will assign a higher precedence to the most specific +one. As an alternative way of defining concept refinements, you can use the +object inheritance syntax involving the `of` keyword: + + ```nim + type + Graph = concept g, type G of EquallyComparable, Copyable + type + VertexType = G.VertexType + EdgeType = G.EdgeType + + VertexType is Copyable + EdgeType is Copyable + + var + v: VertexType + e: EdgeType + + IncidendeGraph = concept of Graph + # symbols such as variables and types from the refined + # concept are automatically in scope: + + g.source(e) is VertexType + g.target(e) is VertexType + + g.outgoingEdges(v) is Enumerable[EdgeType] + + BidirectionalGraph = concept g, type G + # The following will also turn the concept into a refinement when it + # comes to overload resolution, but it doesn't provide the convenient + # symbol inheritance + g is IncidendeGraph + + g.incomingEdges(G.VertexType) is Enumerable[G.EdgeType] + + proc f(g: IncidendeGraph) + proc f(g: BidirectionalGraph) # this one will be preferred if we pass a type + # matching the BidirectionalGraph concept + ``` + +.. + Converter type classes + ---------------------- + + Concepts can also be used to convert a whole range of types to a single type or + a small set of simpler types. This is achieved with a `return` statement within + the concept body: + + ```nim + type + Stringable = concept x + $x is string + return $x + + StringRefValue[CharType] = object + base: ptr CharType + len: int + + StringRef = concept x + # the following would be an overloaded proc for cstring, string, seq and + # other user-defined types, returning either a StringRefValue[char] or + # StringRefValue[wchar] + return makeStringRefValue(x) + + # the varargs param will here be converted to an array of StringRefValues + # the proc will have only two instantiations for the two character types + proc log(format: static string, varargs[StringRef]) + + # this proc will allow char and wchar values to be mixed in + # the same call at the cost of additional instantiations + # the varargs param will be converted to a tuple + proc log(format: static string, varargs[distinct StringRef]) + ``` + + +.. + VTable types + ------------ + + Concepts allow Nim to define a great number of algorithms, using only + static polymorphism and without erasing any type information or sacrificing + any execution speed. But when polymorphic collections of objects are required, + the user must use one of the provided type erasure techniques - either common + base types or VTable types. + + VTable types are represented as "fat pointers" storing a reference to an + object together with a reference to a table of procs implementing a set of + required operations (the so called vtable). + + In contrast to other programming languages, the vtable in Nim is stored + externally to the object, allowing you to create multiple different vtable + views for the same object. Thus, the polymorphism in Nim is unbounded - + any type can implement an unlimited number of protocols or interfaces not + originally envisioned by the type's author. + + Any concept type can be turned into a VTable type by using the `vtref` + or the `vtptr` compiler magics. Under the hood, these magics generate + a converter type class, which converts the regular instances of the matching + types to the corresponding VTable type. + + ```nim + type + IntEnumerable = vtref Enumerable[int] + + MyObject = object + enumerables: seq[IntEnumerable] + streams: seq[OutputStream.vtref] + + proc addEnumerable(o: var MyObject, e: IntEnumerable) = + o.enumerables.add e + + proc addStream(o: var MyObject, e: OutputStream.vtref) = + o.streams.add e + ``` + + The procs that will be included in the vtable are derived from the concept + body and include all proc calls for which all param types were specified as + concrete types. All such calls should include exactly one param of the type + matched against the concept (not necessarily in the first position), which + will be considered the value bound to the vtable. + + Overloads will be created for all captured procs, accepting the vtable type + in the position of the captured underlying object. + + Under these rules, it's possible to obtain a vtable type for a concept with + unbound type parameters or one instantiated with metatypes (type classes), + but it will include a smaller number of captured procs. A completely empty + vtable will be reported as an error. + + The `vtref` magic produces types which can be bound to `ref` types and + the `vtptr` magic produced types bound to `ptr` types. + + +.. + deepCopy + -------- + `=deepCopy` is a builtin that is invoked whenever data is passed to + a `spawn`'ed proc to ensure memory safety. The programmer can override its + behaviour for a specific `ref` or `ptr` type `T`. (Later versions of the + language may weaken this restriction.) + + The signature has to be: + + ```nim + proc `=deepCopy`(x: T): T + ``` + + This mechanism will be used by most data structures that support shared memory, + like channels, to implement thread safe automatic memory management. + + The builtin `deepCopy` can even clone closures and their environments. See + the documentation of [spawn][spawn statement] for details. + + +Dynamic arguments for bindSym +============================= + +This experimental feature allows the symbol name argument of `macros.bindSym` +to be computed dynamically. + + ```nim + {.experimental: "dynamicBindSym".} + + import std/macros + + macro callOp(opName, arg1, arg2): untyped = + result = newCall(bindSym($opName), arg1, arg2) + + echo callOp("+", 1, 2) + echo callOp("-", 5, 4) + ``` + + +Term rewriting macros +===================== + +Term rewriting macros are macros or templates that have not only +a *name* but also a *pattern* that is searched for after the semantic checking +phase of the compiler: This means they provide an easy way to enhance the +compilation pipeline with user defined optimizations: + + ```nim + template optMul{`*`(a, 2)}(a: int): int = a + a + + let x = 3 + echo x * 2 + ``` + +The compiler now rewrites `x * 2` as `x + x`. The code inside the +curly brackets is the pattern to match against. The operators `*`, `**`, +`|`, `~` have a special meaning in patterns if they are written in infix +notation, so to match verbatim against `*` the ordinary function call syntax +needs to be used. + +Term rewriting macros are applied recursively, up to a limit. This means that +if the result of a term rewriting macro is eligible for another rewriting, +the compiler will try to perform it, and so on, until no more optimizations +are applicable. To avoid putting the compiler into an infinite loop, there is +a hard limit on how many times a single term rewriting macro can be applied. +Once this limit has been passed, the term rewriting macro will be ignored. + +Unfortunately optimizations are hard to get right and even this tiny example +is **wrong**: + + ```nim + template optMul{`*`(a, 2)}(a: int): int = a + a + + proc f(): int = + echo "side effect!" + result = 55 + + echo f() * 2 + ``` + +We cannot duplicate 'a' if it denotes an expression that has a side effect! +Fortunately Nim supports side effect analysis: + + ```nim + template optMul{`*`(a, 2)}(a: int{noSideEffect}): int = a + a + + proc f(): int = + echo "side effect!" + result = 55 + + echo f() * 2 # not optimized ;-) + ``` + +You can make one overload matching with a constraint and one without, and the +one with a constraint will have precedence, and so you can handle both cases +differently. + +So what about `2 * a`? We should tell the compiler `*` is commutative. We +cannot really do that however as the following code only swaps arguments +blindly: + + ```nim + template mulIsCommutative{`*`(a, b)}(a, b: int): int = b * a + ``` + +What optimizers really need to do is a *canonicalization*: + + ```nim + template canonMul{`*`(a, b)}(a: int{lit}, b: int): int = b * a + ``` + +The `int{lit}` parameter pattern matches against an expression of +type `int`, but only if it's a literal. + + + +Parameter constraints +--------------------- + +The `parameter constraint`:idx: expression can use the operators `|` (or), +`&` (and) and `~` (not) and the following predicates: + +=================== ===================================================== +Predicate Meaning +=================== ===================================================== +`atom` The matching node has no children. +`lit` The matching node is a literal like `"abc"`, `12`. +`sym` The matching node must be a symbol (a bound + identifier). +`ident` The matching node must be an identifier (an unbound + identifier). +`call` The matching AST must be a call/apply expression. +`lvalue` The matching AST must be an lvalue. +`sideeffect` The matching AST must have a side effect. +`nosideeffect` The matching AST must have no side effect. +`param` A symbol which is a parameter. +`genericparam` A symbol which is a generic parameter. +`module` A symbol which is a module. +`type` A symbol which is a type. +`var` A symbol which is a variable. +`let` A symbol which is a `let` variable. +`const` A symbol which is a constant. +`result` The special `result` variable. +`proc` A symbol which is a proc. +`method` A symbol which is a method. +`iterator` A symbol which is an iterator. +`converter` A symbol which is a converter. +`macro` A symbol which is a macro. +`template` A symbol which is a template. +`field` A symbol which is a field in a tuple or an object. +`enumfield` A symbol which is a field in an enumeration. +`forvar` A for loop variable. +`label` A label (used in `block` statements). +`nk*` The matching AST must have the specified kind. + (Example: `nkIfStmt` denotes an `if` statement.) +`alias` States that the marked parameter needs to alias + with *some* other parameter. +`noalias` States that *every* other parameter must not alias + with the marked parameter. +=================== ===================================================== + +Predicates that share their name with a keyword have to be escaped with +backticks. +The `alias` and `noalias` predicates refer not only to the matching AST, +but also to every other bound parameter; syntactically they need to occur after +the ordinary AST predicates: + + ```nim + template ex{a = b + c}(a: int{noalias}, b, c: int) = + # this transformation is only valid if 'b' and 'c' do not alias 'a': + a = b + inc a, c + ``` + +Another example: + + ```nim + proc somefunc(s: string) = assert s == "variable" + proc somefunc(s: string{nkStrLit}) = assert s == "literal" + proc somefunc(s: string{nkRStrLit}) = assert s == r"raw" + proc somefunc(s: string{nkTripleStrLit}) = assert s == """triple""" + proc somefunc(s: static[string]) = assert s == "constant" + + # Use parameter constraints to provide overloads based on both the input parameter type and form. + var variable = "variable" + somefunc(variable) + const constant = "constant" + somefunc(constant) + somefunc("literal") + somefunc(r"raw") + somefunc("""triple""") + ``` + + +Pattern operators +----------------- + +The operators `*`, `**`, `|`, `~` have a special meaning in patterns +if they are written in infix notation. + + +### The `|` operator + +The `|` operator if used as infix operator creates an ordered choice: + + ```nim + template t{0|1}(): untyped = 3 + let a = 1 + # outputs 3: + echo a + ``` + +The matching is performed after the compiler performed some optimizations like +constant folding, so the following does not work: + + ```nim + template t{0|1}(): untyped = 3 + # outputs 1: + echo 1 + ``` + +The reason is that the compiler already transformed the 1 into "1" for +the `echo` statement. However, a term rewriting macro should not change the +semantics anyway. In fact, they can be deactivated with the `--patterns:off`:option: +command line option or temporarily with the `patterns` pragma. + + +### The `{}` operator + +A pattern expression can be bound to a pattern parameter via the `expr{param}` +notation: + + ```nim + template t{(0|1|2){x}}(x: untyped): untyped = x + 1 + let a = 1 + # outputs 2: + echo a + ``` + + +### The `~` operator + +The `~` operator is the 'not' operator in patterns: + + ```nim + template t{x = (~x){y} and (~x){z}}(x, y, z: bool) = + x = y + if x: x = z + + var + a = false + b = true + c = false + a = b and c + echo a + ``` + + +### The `*` operator + +The `*` operator can *flatten* a nested binary expression like `a & b & c` +to `&(a, b, c)`: + + ```nim + var + calls = 0 + + proc `&&`(s: varargs[string]): string = + result = s[0] + for i in 1..len(s)-1: result.add s[i] + inc calls + + template optConc{ `&&` * a }(a: string): untyped = &&a + + let space = " " + echo "my" && (space & "awe" && "some " ) && "concat" + + # check that it's been optimized properly: + doAssert calls == 1 + ``` + + +The second operator of `*` must be a parameter; it is used to gather all the +arguments. The expression `"my" && (space & "awe" && "some " ) && "concat"` +is passed to `optConc` in `a` as a special list (of kind `nkArgList`) +which is flattened into a call expression; thus the invocation of `optConc` +produces: + + ```nim + `&&`("my", space & "awe", "some ", "concat") + ``` + + +### The `**` operator + +The `**` is much like the `*` operator, except that it gathers not only +all the arguments, but also the matched operators in reverse polish notation: + + ```nim + import std/macros + + type + Matrix = object + dummy: int + + proc `*`(a, b: Matrix): Matrix = discard + proc `+`(a, b: Matrix): Matrix = discard + proc `-`(a, b: Matrix): Matrix = discard + proc `$`(a: Matrix): string = result = $a.dummy + proc mat21(): Matrix = + result.dummy = 21 + + macro optM{ (`+`|`-`|`*`) ** a }(a: Matrix): untyped = + echo treeRepr(a) + result = newCall(bindSym"mat21") + + var x, y, z: Matrix + + echo x + y * z - x + ``` + +This passes the expression `x + y * z - x` to the `optM` macro as +an `nnkArgList` node containing: + + Arglist + Sym "x" + Sym "y" + Sym "z" + Sym "*" + Sym "+" + Sym "x" + Sym "-" + +(This is the reverse polish notation of `x + y * z - x`.) + + +Parameters +---------- + +Parameters in a pattern are type checked in the matching process. If a +parameter is of the type `varargs`, it is treated specially and can match +0 or more arguments in the AST to be matched against: + + ```nim + template optWrite{ + write(f, x) + ((write|writeLine){w})(f, y) + }(x, y: varargs[untyped], f: File, w: untyped) = + w(f, x, y) + ``` + + +noRewrite pragma +---------------- + +Term rewriting macros and templates are currently greedy and +they will rewrite as long as there is a match. +There was no way to ensure some rewrite happens only once, +e.g. when rewriting term to same term plus extra content. + +`noRewrite` pragma can actually prevent further rewriting on marked code, +e.g. with given example `echo("ab")` will be rewritten just once: + + ```nim + template pwnEcho{echo(x)}(x: untyped) = + {.noRewrite.}: echo("pwned!") + + echo "ab" + ``` + +`noRewrite` pragma can be useful to control term-rewriting macros recursion. + + + +Example: Partial evaluation +--------------------------- + +The following example shows how some simple partial evaluation can be +implemented with term rewriting: + + ```nim + proc p(x, y: int; cond: bool): int = + result = if cond: x + y else: x - y + + template optP1{p(x, y, true)}(x, y: untyped): untyped = x + y + template optP2{p(x, y, false)}(x, y: untyped): untyped = x - y + ``` + + +Example: Hoisting +----------------- + +The following example shows how some form of hoisting can be implemented: + + ```nim + import std/pegs + + template optPeg{peg(pattern)}(pattern: string{lit}): Peg = + var gl {.global, gensym.} = peg(pattern) + gl + + for i in 0 .. 3: + echo match("(a b c)", peg"'(' @ ')'") + echo match("W_HI_Le", peg"\y 'while'") + ``` + +The `optPeg` template optimizes the case of a peg constructor with a string +literal, so that the pattern will only be parsed once at program startup and +stored in a global `gl` which is then re-used. This optimization is called +hoisting because it is comparable to classical loop hoisting. + + +AST based overloading +===================== + +Parameter constraints can also be used for ordinary routine parameters; these +constraints then affect ordinary overloading resolution: + + ```nim + proc optLit(a: string{lit|`const`}) = + echo "string literal" + proc optLit(a: string) = + echo "no string literal" + + const + constant = "abc" + + var + variable = "xyz" + + optLit("literal") + optLit(constant) + optLit(variable) + ``` + +However, the constraints `alias` and `noalias` are not available in +ordinary routines. + + +Parallel & Spawn +================ + +Nim has two flavors of parallelism: +1) `Structured`:idx: parallelism via the `parallel` statement. +2) `Unstructured`:idx: parallelism via the standalone `spawn` statement. + +Nim has a builtin thread pool that can be used for CPU intensive tasks. For +IO intensive tasks the `async` and `await` features should be +used instead. Both parallel and spawn need the [threadpool](threadpool.html) +module to work. + +Somewhat confusingly, `spawn` is also used in the `parallel` statement +with slightly different semantics. `spawn` always takes a call expression of +the form `f(a, ...)`. Let `T` be `f`'s return type. If `T` is `void`, +then `spawn`'s return type is also `void`, otherwise it is `FlowVar[T]`. + +Within a `parallel` section, the `FlowVar[T]` is sometimes eliminated +to `T`. This happens when `T` does not contain any GC'ed memory. +The compiler can ensure the location in `location = spawn f(...)` is not +read prematurely within a `parallel` section and so there is no need for +the overhead of an indirection via `FlowVar[T]` to ensure correctness. + +.. note:: Currently exceptions are not propagated between `spawn`'ed tasks! + +This feature is likely to be removed in the future as external packages +can have better solutions. + + +Spawn statement +--------------- + +The `spawn`:idx: statement can be used to pass a task to the thread pool: + + ```nim + import std/threadpool + + proc processLine(line: string) = + discard "do some heavy lifting here" + + for x in lines("myinput.txt"): + spawn processLine(x) + sync() + ``` + +For reasons of type safety and implementation simplicity the expression +that `spawn` takes is restricted: + +* It must be a call expression `f(a, ...)`. +* `f` must be `gcsafe`. +* `f` must not have the calling convention `closure`. +* `f`'s parameters may not be of type `var`. + This means one has to use raw `ptr`'s for data passing reminding the + programmer to be careful. +* `ref` parameters are deeply copied, which is a subtle semantic change and + can cause performance problems, but ensures memory safety. This deep copy + is performed via `system.deepCopy`, so it can be overridden. +* For *safe* data exchange between `f` and the caller, a global `Channel` + needs to be used. However, since spawn can return a result, often no further + communication is required. + + +`spawn` executes the passed expression on the thread pool and returns +a `data flow variable`:idx: `FlowVar[T]` that can be read from. The reading +with the `^` operator is **blocking**. However, one can use `blockUntilAny` to +wait on multiple flow variables at the same time: + + ```nim + import std/threadpool, ... + + # wait until 2 out of 3 servers received the update: + proc main = + var responses = newSeq[FlowVarBase](3) + for i in 0..2: + responses[i] = spawn tellServer(Update, "key", "value") + var index = blockUntilAny(responses) + assert index >= 0 + responses.del(index) + discard blockUntilAny(responses) + ``` + +Data flow variables ensure that no data races are possible. Due to +technical limitations, not every type `T` can be used in +a data flow variable: `T` has to be a `ref`, `string`, `seq` +or of a type that doesn't contain any GC'd type. This +restriction is not hard to work-around in practice. + + + +Parallel statement +------------------ + +Example: + + ```nim test = "nim c --threads:on $1" + # Compute pi in an inefficient way + import std/[strutils, math, threadpool] + {.experimental: "parallel".} + + proc term(k: float): float = 4 * math.pow(-1, k) / (2*k + 1) + + proc pi(n: int): float = + var ch = newSeq[float](n + 1) + parallel: + for k in 0..ch.high: + ch[k] = spawn term(float(k)) + for k in 0..ch.high: + result += ch[k] + + echo formatFloat(pi(5000)) + ``` + + +The parallel statement is the preferred mechanism to introduce parallelism in a +Nim program. Only a subset of the Nim language is valid within a `parallel` +section. This subset is checked during semantic analysis to be free of data +races. A sophisticated `disjoint checker`:idx: ensures that no data races are +possible, even though shared memory is extensively supported! + +The subset is in fact the full language with the following +restrictions / changes: + +* `spawn` within a `parallel` section has special semantics. +* Every location of the form `a[i]`, `a[i..j]` and `dest` where + `dest` is part of the pattern `dest = spawn f(...)` has to be + provably disjoint. This is called the *disjoint check*. +* Every other complex location `loc` that is used in a spawned + proc (`spawn f(loc)`) has to be immutable for the duration of + the `parallel` section. This is called the *immutability check*. Currently + it is not specified what exactly "complex location" means. We need to make + this an optimization! +* Every array access has to be provably within bounds. This is called + the *bounds check*. +* Slices are optimized so that no copy is performed. This optimization is not + yet performed for ordinary slices outside of a `parallel` section. + + +Strict definitions and `out` parameters +======================================= + +With `experimental: "strictDefs"` *every* local variable must be initialized explicitly before it can be used: + + ```nim + {.experimental: "strictDefs".} + + proc test = + var s: seq[string] + s.add "abc" # invalid! + + ``` + +Needs to be written as: + + ```nim + {.experimental: "strictDefs".} + + proc test = + var s: seq[string] = @[] + s.add "abc" # valid! + + ``` + +A control flow analysis is performed in order to prove that a variable has been written to +before it is used. Thus the following is valid: + + ```nim + {.experimental: "strictDefs".} + + proc test(cond: bool) = + var s: seq[string] + if cond: + s = @["y"] + else: + s = @[] + s.add "abc" # valid! + ``` + +In this example every path does set `s` to a value before it is used. + + ```nim + {.experimental: "strictDefs".} + + proc test(cond: bool) = + let s: seq[string] + if cond: + s = @["y"] + else: + s = @[] + ``` + +With `experimental: "strictDefs"`, `let` statements are allowed to not have an initial value, but every path should set `s` to a value before it is used. + + +`out` parameters +---------------- + +An `out` parameter is like a `var` parameter but it must be written to before it can be used: + + ```nim + proc myopen(f: out File; name: string): bool = + f = default(File) + result = open(f, name) + ``` + +While it is usually the better style to use the return type in order to return results API and ABI +considerations might make this infeasible. Like for `var T` Nim maps `out T` to a hidden pointer. +For example POSIX's `stat` routine can be wrapped as: + + ```nim + proc stat*(a1: cstring, a2: out Stat): cint {.importc, header: "<sys/stat.h>".} + ``` + +When the implementation of a routine with output parameters is analysed, the compiler +checks that every path before the (implicit or explicit) return does set every output +parameter: + + ```nim + proc p(x: out int; y: out string; cond: bool) = + x = 4 + if cond: + y = "abc" + # error: not every path initializes 'y' + ``` + + +Out parameters and exception handling +------------------------------------- + +The analysis should take exceptions into account (but currently does not): + + ```nim + proc p(x: out int; y: out string; cond: bool) = + x = canRaise(45) + y = "abc" # <-- error: not every path initializes 'y' + ``` + +Once the implementation takes exceptions into account it is easy enough to +use `outParam = default(typeof(outParam))` in the beginning of the proc body. + +Out parameters and inheritance +------------------------------ + +It is not valid to pass an lvalue of a supertype to an `out T` parameter: + + ```nim + type + Superclass = object of RootObj + a: int + Subclass = object of Superclass + s: string + + proc init(x: out Superclass) = + x = Superclass(a: 8) + + var v: Subclass + init v + use v.s # the 's' field was never initialized! + ``` + +However, in the future this could be allowed and provide a better way to write object +constructors that take inheritance into account. + + +**Note**: The implementation of "strict definitions" and "out parameters" is experimental but the concept +is solid and it is expected that eventually this mode becomes the default in later versions. + + +Strict case objects +=================== + +With `experimental: "strictCaseObjects"` *every* field access is checked to be valid at compile-time. +The field is within a `case` section of an `object`. + + ```nim + {.experimental: "strictCaseObjects".} + + type + Foo = object + case b: bool + of false: + s: string + of true: + x: int + + var x = Foo(b: true, x: 4) + case x.b + of true: + echo x.x # valid + of false: + echo "no" + + case x.b + of false: + echo x.x # error: field access outside of valid case branch: x.x + of true: + echo "no" + + ``` + +**Note**: The implementation of "strict case objects" is experimental but the concept +is solid and it is expected that eventually this mode becomes the default in later versions. + + +Quirky routines +=============== + +The default code generation strategy of exceptions under the ARC/ORC model is the so called +`--exceptions:goto` implementation. This implementation inserts a check after every call that +can potentially raise an exception. A typical instruction sequence for this on +for a x86 64 bit machine looks like: + + ``` + cmp DWORD PTR [rbx], 0 + je .L1 + ``` + +This is a memory fetch followed by jump. (An ideal implementation would +use the carry flag and a single instruction like ``jc .L1``.) + +This overhead might not be desired and depending on the semantics of the routine may not be required +either. +So it can be disabled via a `.quirky` annotation: + + ```nim + proc wontRaise(x: int) {.quirky.} = + if x != 0: + # because of `quirky` this will continue even if `write` raised an IO exception: + write x + wontRaise(x-1) + + wontRaise 10 + + ``` + +If the used exception model is not `--exceptions:goto` then the `quirky` pragma has no effect and is +ignored. + +The `quirky` pragma can also be be pushed in order to affect a group of routines and whether +the compiler supports the pragma can be checked with `defined(nimHasQuirky)`: + + ```nim + when defined(nimHasQuirky): + {.push quirky: on.} + + proc doRaise() = raise newException(ValueError, "") + + proc f(): string = "abc" + + proc q(cond: bool) = + if cond: + doRaise() + echo f() + + q(true) + + when defined(nimHasQuirky): + {.pop.} + ``` + +**Warning**: The `quirky` pragma only affects code generation, no check for validity is performed! + + +Threading under ARC/ORC +======================= + +ARC/ORC supports a shared heap out of the box. This means that messages can be sent between +threads without copies. However, without copying the data there is an inherent danger of +data races. Data races are prevented at compile-time if it is enforced that +only **isolated** subgraphs can be sent around. + + +Isolation +--------- + +The standard library module `isolation.nim` provides a generic type `Isolated[T]` that +captures the important notion that nothing else can reference the graph that is wrapped +inside `Isolated[T]`. It is what a channel implementation should use in order to enforce +the freedom of data races: + + ```nim + proc send*[T](c: var Channel[T]; msg: sink Isolated[T]) + proc recv*[T](c: var Channel[T]): T + ## Note: Returns T, not Isolated[T] for convenience. + + proc recvIso*[T](c: var Channel[T]): Isolated[T] + ## remembers the data is Isolated[T]. + ``` + +In order to create an `Isolated` graph one has to use either `isolate` or `unsafeIsolate`. +`unsafeIsolate` is as its name says unsafe and no checking is performed. It should be considered +to be as dangerous as a `cast` operation. + + +Construction must ensure that the invariant holds, namely that the wrapped `T` +is free of external aliases into it. `isolate` ensures this invariant. It is +inspired by Pony's `recover` construct: + + ```nim + func isolate(x: sink T): Isolated[T] {.magic: "Isolate".} + ``` + + +As you can see, this is a new builtin because the check it performs on `x` is non-trivial: + +If `T` does not contain a `ref` or `closure` type, it is isolated. Else the syntactic +structure of `x` is analyzed: + +- Literals like `nil`, `4`, `"abc"` are isolated. +- A local variable or a routine parameter is isolated if either of these conditions is true: + 1. Its type is annotated with the `.sendable` pragma. Note `Isolated[T]` is annotated as + `.sendable`. + 2. Its type contains the potentially dangerous `ref` and `proc {.closure}` types + only in places that are protected via a `.sendable` container. + +- An array constructor `[x...]` is isolated if every element `x` is isolated. +- An object constructor `Obj(x...)` is isolated if every element `x` is isolated. +- An `if` or `case` expression is isolated if all possible values the expression + may return are isolated. +- A type conversion `C(x)` is isolated if `x` is isolated. Analogous for `cast` + expressions. +- A function call `f(x...)` is isolated if `f` is `.noSideEffect` and for every argument `x`: + - `x` is isolated **or** + - `f`'s return type cannot *alias* `x`'s type. This is checked via a form of alias analysis as explained in the next paragraph. + + + +Alias analysis +-------------- + +We start with an important, simple case that must be valid: Sending the result +of `parseJson` to a channel. Since the signature +is `func parseJson(input: string): JsonNode` it is easy to see that JsonNode +can never simply be a view into `input` which is a `string`. + +A different case is the identity function `id`, `send id(myJsonGraph)` must be +invalid because we do not know how many aliases into `myJsonGraph` exist +elsewhere. + +In general type `A` can alias type `T` if: + +- `A` and `T` are the same types. +- `A` is a distinct type derived from `T`. +- `A` is a field inside `T` if `T` is a final object type. +- `T` is an inheritable object type. (An inherited type could always contain + a `field: A`). +- `T` is a closure type. Reason: `T`'s environment can contain a field of + type `A`. +- `A` is the element type of `T` if `T` is an array, sequence or pointer type. + + + + +Sendable pragma +--------------- + +A container type can be marked as `.sendable`. `.sendable` declares that the type +encapsulates a `ref` type effectively so that a variable of this container type +can be used in an `isolate` context: + + ```nim + type + Isolated*[T] {.sendable.} = object ## Isolated data can only be moved, not copied. + value: T + + proc `=copy`*[T](dest: var Isolated[T]; src: Isolated[T]) {.error.} + + proc `=sink`*[T](dest: var Isolated[T]; src: Isolated[T]) {.inline.} = + # delegate to value's sink operation + `=sink`(dest.value, src.value) + + proc `=destroy`*[T](dest: var Isolated[T]) {.inline.} = + # delegate to value's destroy operation + `=destroy`(dest.value) + ``` + +The `.sendable` pragma itself is an experimenal, unchecked, unsafe annotation. It is +currently only used by `Isolated[T]`. + +Virtual pragma +============== + +`virtual` is designed to extend or create virtual functions when targeting the cpp backend. When a proc is marked with virtual, it forward declares the proc header within the type's body. + +Here's an example of how to use the virtual pragma: + +```nim +proc newCpp*[T](): ptr T {.importcpp: "new '*0()".} +type + Foo = object of RootObj + FooPtr = ptr Foo + Boo = object of Foo + BooPtr = ptr Boo + +proc salute(self: FooPtr) {.virtual.} = + echo "hello foo" + +proc salute(self: BooPtr) {.virtual.} = + echo "hello boo" + +let foo = newCpp[Foo]() +let boo = newCpp[Boo]() +let booAsFoo = cast[FooPtr](newCpp[Boo]()) + +foo.salute() # prints hello foo +boo.salute() # prints hello boo +booAsFoo.salute() # prints hello boo +``` +In this example, the `salute` function is virtual in both Foo and Boo types. This allows for polymorphism. + +The virtual pragma also supports a special syntax to express Cpp constraints. Here's how it works: + +`$1` refers to the function name +`'idx` refers to the type of the argument at the position idx. Where idx = 1 is the `this` argument. +`#idx` refers to the argument name. + +The return type can be referred to as `-> '0`, but this is optional and often not needed. + + ```nim + {.emit:"""/*TYPESECTION*/ +#include <iostream> + class CppPrinter { + public: + + virtual void printConst(char* message) const { + std::cout << "Const Message: " << message << std::endl; + } + virtual void printConstRef(char* message, const int& flag) const { + std::cout << "Const Ref Message: " << message << std::endl; + } +}; +""".} + +type + CppPrinter {.importcpp, inheritable.} = object + NimPrinter {.exportc.} = object of CppPrinter + +proc printConst(self: CppPrinter; message:cstring) {.importcpp.} +CppPrinter().printConst(message) + +# override is optional. +proc printConst(self: NimPrinter; message: cstring) {.virtual: "$1('2 #2) const override".} = + echo "NimPrinter: " & $message + +proc printConstRef(self: NimPrinter; message: cstring; flag:int32) {.virtual: "$1('2 #2, const '3& #3 ) const override".} = + echo "NimPrinterConstRef: " & $message + +NimPrinter().printConst(message) +var val: int32 = 10 +NimPrinter().printConstRef(message, val) + +``` + +Constructor pragma +================== + +The `constructor` pragma can be used in two ways: in conjunction with `importcpp` to import a C++ constructor, and to declare constructors that operate similarly to `virtual`. + +Consider: + +```nim +type Foo* = object + x: int32 + +proc makeFoo(x: int32): Foo {.constructor.} = + result.x = x +``` + +It forward declares the constructor in the type definition. When the constructor has parameters, it also generates a default constructor. One can avoid this behaviour by using `noDecl` in a default constructor. + +Like `virtual`, `constructor` also supports a syntax that allows to express C++ constraints. + +For example: + +```nim +{.emit:"""/*TYPESECTION*/ +struct CppClass { + int x; + int y; + CppClass(int inX, int inY) { + this->x = inX; + this->y = inY; + } + //CppClass() = default; +}; +""".} + +type + CppClass* {.importcpp, inheritable.} = object + x: int32 + y: int32 + NimClass* = object of CppClass + +proc makeNimClass(x: int32): NimClass {.constructor:"NimClass('1 #1) : CppClass(0, #1)".} = + result.x = x + +# Optional: define the default constructor explicitly +proc makeCppClass(): NimClass {.constructor: "NimClass() : CppClass(0, 0)".} = + result.x = 1 +``` + +In the example above `CppClass` has a deleted default constructor. Notice how by using the constructor syntax, one can call the appropriate constructor. + +Notice when calling a constructor in the section of a global variable initialization, it will be called before `NimMain` meaning Nim is not fully initialized. + +Constructor Initializer +======================= + +By default Nim initializes `importcpp` types with `{}`. This can be problematic when importing +types with a deleted default constructor. In order to avoid this, one can specify default values for a constructor by specifying default values for the proc params in the `constructor` proc. + +For example: + +```nim + +{.emit: """/*TYPESECTION*/ +struct CppStruct { + CppStruct(int x, char* y): x(x), y(y){} + int x; + char* y; +}; +""".} +type + CppStruct {.importcpp, inheritable.} = object + +proc makeCppStruct(a: cint = 5, b:cstring = "hello"): CppStruct {.importcpp: "CppStruct(@)", constructor.} + +(proc (s: CppStruct) = echo "hello")(makeCppStruct()) +# If one removes a default value from the constructor and passes it to the call explicitly, the C++ compiler will complain. + +``` +Skip initializers in fields members +=================================== + +By using `noInit` in a type or field declaration, the compiler will skip the initializer. By doing so one can explicitly initialize those values in the constructor of the type owner. + +For example: + +```nim + +{.emit: """/*TYPESECTION*/ + struct Foo { + Foo(int a){}; + }; + struct Boo { + Boo(int a){}; + }; + + """.} + +type + Foo {.importcpp.} = object + Boo {.importcpp, noInit.} = object + Test {.exportc.} = object + foo {.noInit.}: Foo + boo: Boo + +proc makeTest(): Test {.constructor: "Test() : foo(10), boo(1)".} = + discard + +proc main() = + var t = makeTest() + +main() + +``` + +Will produce: + +```cpp + +struct Test { + Foo foo; + Boo boo; + N_LIB_PRIVATE N_NOCONV(, Test)(void); +}; + +``` + +Notice that without `noInit` it would produce `Foo foo {}` and `Boo boo {}` + + +Member pragma +============= + +Like the `constructor` and `virtual` pragmas, the `member` pragma can be used to attach a procedure to a C++ type. It's more flexible than the `virtual` pragma in the sense that it accepts not only names but also operators and destructors. + +For example: + +```nim +proc print(s: cstring) {.importcpp: "printf(@)", header: "<stdio.h>".} + +type + Doo {.exportc.} = object + test: int + +proc memberProc(f: Doo) {.member.} = + echo $f.test + +proc destructor(f: Doo) {.member: "~'1()", used.} = + print "destructing\n" + +proc `==`(self, other: Doo): bool {.member: "operator==('2 const & #2) const -> '0".} = + self.test == other.test + +let doo = Doo(test: 2) +doo.memberProc() +echo doo == Doo(test: 1) + +``` + +Will print: +``` +2 +false +destructing +destructing +``` + +Notice how the C++ destructor is called automatically. Also notice the double implementation of `==` as an operator in Nim but also in C++. This is useful if you need the type to match some C++ `concept` or `trait` when interoping. + +A side effect of being able to declare C++ operators, is that you can now also create a +C++ functor to have seamless interop with C++ lambdas (syntactic sugar for functors). + +For example: + +```nim +type + NimFunctor = object + discard +proc invoke(f: NimFunctor; n: int) {.member: "operator ()('2 #2)".} = + echo "FunctorSupport!" + +{.experimental: "callOperator".} +proc `()`(f: NimFunctor; n:int) {.importcpp: "#(@)" .} +NimFunctor()(1) +``` +Notice we use the overload of `()` to have the same semantics in Nim, but on the `importcpp` we import the functor as a function. +This allows to easy interop with functions that accepts for example a `const` operator in its signature. + + +Injected symbols in generic procs and templates +=============================================== + +With the experimental option `openSym`, captured symbols in generic routine and +template bodies may be replaced by symbols injected locally by templates/macros +at instantiation time. `bind` may be used to keep the captured symbols over the +injected ones regardless of enabling the options, but other methods like +renaming the captured symbols should be used instead so that the code is not +affected by context changes. + +Since this change may affect runtime behavior, the experimental switch +`openSym` needs to be enabled; and a warning is given in the case where an +injected symbol would replace a captured symbol not bound by `bind` and +the experimental switch isn't enabled. + +```nim +const value = "captured" +template foo(x: int, body: untyped): untyped = + let value {.inject.} = "injected" + body + +proc old[T](): string = + foo(123): + return value # warning: a new `value` has been injected, use `bind` or turn on `experimental:openSym` +echo old[int]() # "captured" + +template oldTempl(): string = + block: + foo(123): + value # warning: a new `value` has been injected, use `bind` or turn on `experimental:openSym` +echo oldTempl() # "captured" + +{.experimental: "openSym".} + +proc bar[T](): string = + foo(123): + return value +assert bar[int]() == "injected" # previously it would be "captured" + +proc baz[T](): string = + bind value + foo(123): + return value +assert baz[int]() == "captured" + +template barTempl(): string = + block: + foo(123): + value +assert barTempl() == "injected" # previously it would be "captured" + +template bazTempl(): string = + bind value + block: + foo(123): + value +assert bazTempl() == "captured" +``` + +This option also generates a new node kind `nnkOpenSym` which contains +exactly 1 `nnkSym` node. In the future this might be merged with a slightly +modified `nnkOpenSymChoice` node but macros that want to support the +experimental feature should still handle `nnkOpenSym`, as the node kind would +simply not be generated as opposed to being removed. + +Another experimental switch `genericsOpenSym` exists that enables this behavior +at instantiation time, meaning templates etc can enable it specifically when +they are being called. However this does not generate `nnkOpenSym` nodes +(unless the other switch is enabled) and so doesn't reflect the regular +behavior of the switch. + +```nim +const value = "captured" +template foo(x: int, body: untyped): untyped = + let value {.inject.} = "injected" + {.push experimental: "genericsOpenSym".} + body + {.pop.} + +proc bar[T](): string = + foo(123): + return value +echo bar[int]() # "injected" + +template barTempl(): string = + block: + var res: string + foo(123): + res = value + res +assert barTempl() == "injected" +``` + + +VTable for methods +================== + +Methods now support implementations based on a VTable by using `--experimental:vtables`. Note that the option needs to enabled +globally. The virtual method table is stored in the type info of +an object, which is an array of function pointers. + +```nim +method foo(x: Base, ...) {.base.} +method foo(x: Derived, ...) {.base.} +``` + +It roughly generates a dispatcher like + +```nim +proc foo_dispatch(x: Base, ...) = + x.typeinfo.vtable[method_index](x, ...) # method_index is the index of the sorted order of a method +``` + +Methods are required to be in the same module where their type has been defined. + +```nim +# types.nim +type + Base* = ref object +``` + +```nim +import types + +method foo(x: Base) {.base.} = discard +``` + +It gives an error: method `foo` can be defined only in the same module with its type (Base). + + +asmSyntax pragma +================ + +The `asmSyntax` pragma is used to specify target inline assembler syntax in an `asm` statement. + +It prevents compiling code with different of the target CC inline asm syntax, i.e. it will not allow gcc inline asm code to be compiled with vcc. + +```nim +proc nothing() = + asm {.asmSyntax: "gcc".}""" + nop + """ +``` + +The current C(C++) backend implementation cannot generate code for gcc and for vcc at the same time. For example, `{.asmSyntax: "vcc".}` with the ICC compiler will not generate code with intel asm syntax, even though ICC can use both gcc-like and vcc-like asm. diff --git a/doc/manual_experimental.rst b/doc/manual_experimental.rst deleted file mode 100644 index b0614885a..000000000 --- a/doc/manual_experimental.rst +++ /dev/null @@ -1,1846 +0,0 @@ -========================= -Nim Experimental Features -========================= - -:Authors: Andreas Rumpf -:Version: |nimversion| - -.. contents:: - - -About this document -=================== - -This document describes features of Nim that are to be considered experimental. -Some of these are not covered by the ``.experimental`` pragma or -``--experimental`` switch because they are already behind a special syntax and -one may want to use Nim libraries using these features without using them -oneself. - -**Note**: Unless otherwise indicated, these features are not to be removed, -but refined and overhauled. - - -Package level objects -===================== - -Every Nim module resides in a (nimble) package. An object type can be attached -to the package it resides in. If that is done, the type can be referenced from -other modules as an `incomplete`:idx: object type. This feature allows to -break up recursive type dependencies across module boundaries. Incomplete -object types are always passed ``byref`` and can only be used in pointer like -contexts (``var/ref/ptr IncompleteObject``) in general since the compiler does -not yet know the size of the object. To complete an incomplete object -the ``package`` pragma has to be used. ``package`` implies ``byref``. - -As long as a type ``T`` is incomplete, neither ``sizeof(T)`` nor runtime -type information for ``T`` is available. - - -Example: - -.. code-block:: nim - - # module A (in an arbitrary package) - type - Pack.SomeObject = object ## declare as incomplete object of package 'Pack' - Triple = object - a, b, c: ref SomeObject ## pointers to incomplete objects are allowed - - ## Incomplete objects can be used as parameters: - proc myproc(x: SomeObject) = discard - - -.. code-block:: nim - - # module B (in package "Pack") - type - SomeObject* {.package.} = object ## Use 'package' to complete the object - s, t: string - x, y: int - - -Void type -========= - -The ``void`` type denotes the absence of any type. Parameters of -type ``void`` are treated as non-existent, ``void`` as a return type means that -the procedure does not return a value: - -.. code-block:: nim - proc nothing(x, y: void): void = - echo "ha" - - nothing() # writes "ha" to stdout - -The ``void`` type is particularly useful for generic code: - -.. code-block:: nim - proc callProc[T](p: proc (x: T), x: T) = - when T is void: - p() - else: - p(x) - - proc intProc(x: int) = discard - proc emptyProc() = discard - - callProc[int](intProc, 12) - callProc[void](emptyProc) - -However, a ``void`` type cannot be inferred in generic code: - -.. code-block:: nim - callProc(emptyProc) - # Error: type mismatch: got (proc ()) - # but expected one of: - # callProc(p: proc (T), x: T) - -The ``void`` type is only valid for parameters and return types; other symbols -cannot have the type ``void``. - - - -Covariance -========== - -Covariance in Nim can be introduced only through pointer-like types such -as ``ptr`` and ``ref``. Sequence, Array and OpenArray types, instantiated -with pointer-like types will be considered covariant if and only if they -are also immutable. The introduction of a ``var`` modifier or additional -``ptr`` or ``ref`` indirections would result in invariant treatment of -these types. - -``proc`` types are currently always invariant, but future versions of Nim -may relax this rule. - -User-defined generic types may also be covariant with respect to some of -their parameters. By default, all generic params are considered invariant, -but you may choose the apply the prefix modifier ``in`` to a parameter to -make it contravariant or ``out`` to make it covariant: - -.. code-block:: nim - type - AnnotatedPtr[out T] = - metadata: MyTypeInfo - p: ref T - - RingBuffer[out T] = - startPos: int - data: seq[T] - - Action {.importcpp: "std::function<void ('0)>".} [in T] = object - -When the designated generic parameter is used to instantiate a pointer-like -type as in the case of `AnnotatedPtr` above, the resulting generic type will -also have pointer-like covariance: - -.. code-block:: nim - type - GuiWidget = object of RootObj - Button = object of GuiWidget - ComboBox = object of GuiWidget - - var - widgetPtr: AnnotatedPtr[GuiWidget] - buttonPtr: AnnotatedPtr[Button] - - ... - - proc drawWidget[T](x: AnnotatedPtr[GuiWidget]) = ... - - # you can call procs expecting base types by supplying a derived type - drawWidget(buttonPtr) - - # and you can convert more-specific pointer types to more general ones - widgetPtr = buttonPtr - -Just like with regular pointers, covariance will be enabled only for immutable -values: - -.. code-block:: nim - proc makeComboBox[T](x: var AnnotatedPtr[GuiWidget]) = - x.p = new(ComboBox) - - makeComboBox(buttonPtr) # Error, AnnotatedPtr[Button] cannot be modified - # to point to a ComboBox - -On the other hand, in the `RingBuffer` example above, the designated generic -param is used to instantiate the non-pointer ``seq`` type, which means that -the resulting generic type will have covariance that mimics an array or -sequence (i.e. it will be covariant only when instantiated with ``ptr`` and -``ref`` types): - -.. code-block:: nim - - type - Base = object of RootObj - Derived = object of Base - - proc consumeBaseValues(b: RingBuffer[Base]) = ... - - var derivedValues: RingBuffer[Derived] - - consumeBaseValues(derivedValues) # Error, Base and Derived values may differ - # in size - - proc consumeBasePointers(b: RingBuffer[ptr Base]) = ... - - var derivedPointers: RingBuffer[ptr Derived] - - consumeBaseValues(derivedPointers) # This is legal - -Please note that Nim will treat the user-defined pointer-like types as -proper alternatives to the built-in pointer types. That is, types such -as `seq[AnnotatedPtr[T]]` or `RingBuffer[AnnotatedPtr[T]]` will also be -considered covariant and you can create new pointer-like types by instantiating -other user-defined pointer-like types. - -The contravariant parameters introduced with the ``in`` modifier are currently -useful only when interfacing with imported types having such semantics. - - -Automatic dereferencing -======================= - -Automatic dereferencing is performed for the first argument of a routine call. -This feature has to be only enabled via ``{.experimental: "implicitDeref".}``: - -.. code-block:: nim - {.experimental: "implicitDeref".} - - proc depth(x: NodeObj): int = ... - - var - n: Node - new(n) - echo n.depth - # no need to write n[].depth either - -Code reordering -=============== - -The code reordering feature can implicitly rearrange procedure, template, and -macro definitions along with variable declarations and initializations at the top -level scope so that, to a large extent, a programmer should not have to worry -about ordering definitions correctly or be forced to use forward declarations to -preface definitions inside a module. - -.. - NOTE: The following was documentation for the code reordering precursor, - which was {.noForward.}. - - In this mode, procedure definitions may appear out of order and the compiler - will postpone their semantic analysis and compilation until it actually needs - to generate code using the definitions. In this regard, this mode is similar - to the modus operandi of dynamic scripting languages, where the function - calls are not resolved until the code is executed. Here is the detailed - algorithm taken by the compiler: - - 1. When a callable symbol is first encountered, the compiler will only note - the symbol callable name and it will add it to the appropriate overload set - in the current scope. At this step, it won't try to resolve any of the type - expressions used in the signature of the symbol (so they can refer to other - not yet defined symbols). - - 2. When a top level call is encountered (usually at the very end of the - module), the compiler will try to determine the actual types of all of the - symbols in the matching overload set. This is a potentially recursive process - as the signatures of the symbols may include other call expressions, whose - types will be resolved at this point too. - - 3. Finally, after the best overload is picked, the compiler will start - compiling the body of the respective symbol. This in turn will lead the - compiler to discover more call expressions that need to be resolved and steps - 2 and 3 will be repeated as necessary. - - Please note that if a callable symbol is never used in this scenario, its - body will never be compiled. This is the default behavior leading to best - compilation times, but if exhaustive compilation of all definitions is - required, using ``nim check`` provides this option as well. - -Example: - -.. code-block:: nim - - {.experimental: "codeReordering".} - - proc foo(x: int) = - bar(x) - - proc bar(x: int) = - echo(x) - - foo(10) - -Variables can also be reordered as well. Variables that are *initialized* (i.e. -variables that have their declaration and assignment combined in a single -statement) can have their entire initialization statement reordered. Be wary of -what code is executed at the top level: - -.. code-block:: nim - {.experimental: "codeReordering".} - - proc a() = - echo(foo) - - var foo = 5 - - a() # outputs: "5" - -.. - TODO: Let's table this for now. This is an *experimental feature* and so the - specific manner in which ``declared`` operates with it can be decided in - eventuality, because right now it works a bit weirdly. - - The values of expressions involving ``declared`` are decided *before* the - code reordering process, and not after. As an example, the output of this - code is the same as it would be with code reordering disabled. - - .. code-block:: nim - {.experimental: "codeReordering".} - - proc x() = - echo(declared(foo)) - - var foo = 4 - - x() # "false" - -It is important to note that reordering *only* works for symbols at top level -scope. Therefore, the following will *fail to compile:* - -.. code-block:: nim - {.experimental: "codeReordering".} - - proc a() = - b() - proc b() = - echo("Hello!") - - a() - - -Automatic self insertions -========================= - -**Note**: The ``.this`` pragma is deprecated and should not be used anymore. - -Starting with version 0.14 of the language, Nim supports ``field`` as a -shortcut for ``self.field`` comparable to the `this`:idx: keyword in Java -or C++. This feature has to be explicitly enabled via a ``{.this: self.}`` -statement pragma (instead of ``self`` any other identifier can be used too). -This pragma is active for the rest of the module: - -.. code-block:: nim - type - Parent = object of RootObj - parentField: int - Child = object of Parent - childField: int - - {.this: self.} - proc sumFields(self: Child): int = - result = parentField + childField - # is rewritten to: - # result = self.parentField + self.childField - -In addition to fields, routine applications are also rewritten, but only -if no other interpretation of the call is possible: - -.. code-block:: nim - proc test(self: Child) = - echo childField, " ", sumFields() - # is rewritten to: - echo self.childField, " ", sumFields(self) - # but NOT rewritten to: - echo self, self.childField, " ", sumFields(self) - - -Named argument overloading -========================== - -Routines with the same type signature can be called differently if a parameter -has different names. This does not need an ``experimental`` switch, but is an -unstable feature. - -.. code-block::nim - proc foo(x: int) = - echo "Using x: ", x - proc foo(y: int) = - echo "Using y: ", y - - foo(x = 2) - # Using x: 2 - foo(y = 2) - # Using y: 2 - - -Do notation -=========== - -As a special more convenient notation, proc expressions involved in procedure -calls can use the ``do`` keyword: - -.. code-block:: nim - sort(cities) do (x,y: string) -> int: - cmp(x.len, y.len) - - # Less parenthesis using the method plus command syntax: - cities = cities.map do (x:string) -> string: - "City of " & x - - # In macros, the do notation is often used for quasi-quoting - macroResults.add quote do: - if not `ex`: - echo `info`, ": Check failed: ", `expString` - -``do`` is written after the parentheses enclosing the regular proc params. -The proc expression represented by the do block is appended to them. -In calls using the command syntax, the do block will bind to the immediately -preceding expression, transforming it in a call. - -``do`` with parentheses is an anonymous ``proc``; however a ``do`` without -parentheses is just a block of code. The ``do`` notation can be used to -pass multiple blocks to a macro: - -.. code-block:: nim - macro performWithUndo(task, undo: untyped) = ... - - performWithUndo do: - # multiple-line block of code - # to perform the task - do: - # code to undo it - - -Special Operators -================= - -dot operators -------------- - -**Note**: Dot operators are still experimental and so need to be enabled -via ``{.experimental: "dotOperators".}``. - -Nim offers a special family of dot operators that can be used to -intercept and rewrite proc call and field access attempts, referring -to previously undeclared symbol names. They can be used to provide a -fluent interface to objects lying outside the static confines of the -type system such as values from dynamic scripting languages -or dynamic file formats such as JSON or XML. - -When Nim encounters an expression that cannot be resolved by the -standard overload resolution rules, the current scope will be searched -for a dot operator that can be matched against a re-written form of -the expression, where the unknown field or proc name is passed to -an ``untyped`` parameter: - -.. code-block:: nim - a.b # becomes `.`(a, b) - a.b(c, d) # becomes `.`(a, b, c, d) - -The matched dot operators can be symbols of any callable kind (procs, -templates and macros), depending on the desired effect: - -.. code-block:: nim - template `.` (js: PJsonNode, field: untyped): JSON = js[astToStr(field)] - - var js = parseJson("{ x: 1, y: 2}") - echo js.x # outputs 1 - echo js.y # outputs 2 - -The following dot operators are available: - -operator `.` ------------- -This operator will be matched against both field accesses and method calls. - -operator `.()` ---------------- -This operator will be matched exclusively against method calls. It has higher -precedence than the `.` operator and this allows one to handle expressions like -`x.y` and `x.y()` differently if one is interfacing with a scripting language -for example. - -operator `.=` -------------- -This operator will be matched against assignments to missing fields. - -.. code-block:: nim - a.b = c # becomes `.=`(a, b, c) - - -Not nil annotation -================== - -**Note:** This is an experimental feature. It can be enabled with -``{.experimental: "notnil"}``. - -All types for which ``nil`` is a valid value can be annotated with the ``not -nil`` annotation to exclude ``nil`` as a valid value: - -.. code-block:: nim - {.experimental: "notnil"} - - type - PObject = ref TObj not nil - TProc = (proc (x, y: int)) not nil - - proc p(x: PObject) = - echo "not nil" - - # compiler catches this: - p(nil) - - # and also this: - var x: PObject - p(x) - -The compiler ensures that every code path initializes variables which contain -non-nilable pointers. The details of this analysis are still to be specified -here. - - -Concepts -======== - -Concepts, also known as "user-defined type classes", are used to specify an -arbitrary set of requirements that the matched type must satisfy. - -Concepts are written in the following form: - -.. code-block:: nim - type - Comparable = concept x, y - (x < y) is bool - - Stack[T] = concept s, var v - s.pop() is T - v.push(T) - - s.len is Ordinal - - for value in s: - value is T - -The concept is a match if: - -a) all of the expressions within the body can be compiled for the tested type -b) all statically evaluable boolean expressions in the body must be true - -The identifiers following the ``concept`` keyword represent instances of the -currently matched type. You can apply any of the standard type modifiers such -as ``var``, ``ref``, ``ptr`` and ``static`` to denote a more specific type of -instance. You can also apply the `type` modifier to create a named instance of -the type itself: - -.. code-block:: nim - type - MyConcept = concept x, var v, ref r, ptr p, static s, type T - ... - -Within the concept body, types can appear in positions where ordinary values -and parameters are expected. This provides a more convenient way to check for -the presence of callable symbols with specific signatures: - -.. code-block:: nim - type - OutputStream = concept var s - s.write(string) - -In order to check for symbols accepting ``type`` params, you must prefix -the type with the explicit ``type`` modifier. The named instance of the -type, following the ``concept`` keyword is also considered to have the -explicit modifier and will be matched only as a type. - -.. code-block:: nim - type - # Let's imagine a user-defined casting framework with operators - # such as `val.to(string)` and `val.to(JSonValue)`. We can test - # for these with the following concept: - MyCastables = concept x - x.to(type string) - x.to(type JSonValue) - - # Let's define a couple of concepts, known from Algebra: - AdditiveMonoid* = concept x, y, type T - x + y is T - T.zero is T # require a proc such as `int.zero` or 'Position.zero' - - AdditiveGroup* = concept x, y, type T - x is AdditiveMonoid - -x is T - x - y is T - -Please note that the ``is`` operator allows one to easily verify the precise -type signatures of the required operations, but since type inference and -default parameters are still applied in the concept body, it's also possible -to describe usage protocols that do not reveal implementation details. - -Much like generics, concepts are instantiated exactly once for each tested type -and any static code included within the body is executed only once. - - -Concept diagnostics -------------------- - -By default, the compiler will report the matching errors in concepts only when -no other overload can be selected and a normal compilation error is produced. -When you need to understand why the compiler is not matching a particular -concept and, as a result, a wrong overload is selected, you can apply the -``explain`` pragma to either the concept body or a particular call-site. - -.. code-block:: nim - type - MyConcept {.explain.} = concept ... - - overloadedProc(x, y, z) {.explain.} - -This will provide Hints in the compiler output either every time the concept is -not matched or only on the particular call-site. - - -Generic concepts and type binding rules ---------------------------------------- - -The concept types can be parametric just like the regular generic types: - -.. code-block:: nim - ### matrixalgo.nim - - import typetraits - - type - AnyMatrix*[R, C: static int; T] = concept m, var mvar, type M - M.ValueType is T - M.Rows == R - M.Cols == C - - m[int, int] is T - mvar[int, int] = T - - type TransposedType = stripGenericParams(M)[C, R, T] - - AnySquareMatrix*[N: static int, T] = AnyMatrix[N, N, T] - - AnyTransform3D* = AnyMatrix[4, 4, float] - - proc transposed*(m: AnyMatrix): m.TransposedType = - for r in 0 ..< m.R: - for c in 0 ..< m.C: - result[r, c] = m[c, r] - - proc determinant*(m: AnySquareMatrix): int = - ... - - proc setPerspectiveProjection*(m: AnyTransform3D) = - ... - - -------------- - ### matrix.nim - - type - Matrix*[M, N: static int; T] = object - data: array[M*N, T] - - proc `[]`*(M: Matrix; m, n: int): M.T = - M.data[m * M.N + n] - - proc `[]=`*(M: var Matrix; m, n: int; v: M.T) = - M.data[m * M.N + n] = v - - # Adapt the Matrix type to the concept's requirements - template Rows*(M: typedesc[Matrix]): int = M.M - template Cols*(M: typedesc[Matrix]): int = M.N - template ValueType*(M: typedesc[Matrix]): typedesc = M.T - - ------------- - ### usage.nim - - import matrix, matrixalgo - - var - m: Matrix[3, 3, int] - projectionMatrix: Matrix[4, 4, float] - - echo m.transposed.determinant - setPerspectiveProjection projectionMatrix - -When the concept type is matched against a concrete type, the unbound type -parameters are inferred from the body of the concept in a way that closely -resembles the way generic parameters of callable symbols are inferred on -call sites. - -Unbound types can appear both as params to calls such as `s.push(T)` and -on the right-hand side of the ``is`` operator in cases such as `x.pop is T` -and `x.data is seq[T]`. - -Unbound static params will be inferred from expressions involving the `==` -operator and also when types dependent on them are being matched: - -.. code-block:: nim - type - MatrixReducer[M, N: static int; T] = concept x - x.reduce(SquareMatrix[N, T]) is array[M, int] - -The Nim compiler includes a simple linear equation solver, allowing it to -infer static params in some situations where integer arithmetic is involved. - -Just like in regular type classes, Nim discriminates between ``bind once`` -and ``bind many`` types when matching the concept. You can add the ``distinct`` -modifier to any of the otherwise inferable types to get a type that will be -matched without permanently inferring it. This may be useful when you need -to match several procs accepting the same wide class of types: - -.. code-block:: nim - type - Enumerable[T] = concept e - for v in e: - v is T - - type - MyConcept = concept o - # this could be inferred to a type such as Enumerable[int] - o.foo is distinct Enumerable - - # this could be inferred to a different type such as Enumerable[float] - o.bar is distinct Enumerable - - # it's also possible to give an alias name to a `bind many` type class - type Enum = distinct Enumerable - o.baz is Enum - -On the other hand, using ``bind once`` types allows you to test for equivalent -types used in multiple signatures, without actually requiring any concrete -types, thus allowing you to encode implementation-defined types: - -.. code-block:: nim - type - MyConcept = concept x - type T1 = auto - x.foo(T1) - x.bar(T1) # both procs must accept the same type - - type T2 = seq[SomeNumber] - x.alpha(T2) - x.omega(T2) # both procs must accept the same type - # and it must be a numeric sequence - -As seen in the previous examples, you can refer to generic concepts such as -`Enumerable[T]` just by their short name. Much like the regular generic types, -the concept will be automatically instantiated with the bind once auto type -in the place of each missing generic param. - -Please note that generic concepts such as `Enumerable[T]` can be matched -against concrete types such as `string`. Nim doesn't require the concept -type to have the same number of parameters as the type being matched. -If you wish to express a requirement towards the generic parameters of -the matched type, you can use a type mapping operator such as `genericHead` -or `stripGenericParams` within the body of the concept to obtain the -uninstantiated version of the type, which you can then try to instantiate -in any required way. For example, here is how one might define the classic -`Functor` concept from Haskell and then demonstrate that Nim's `Option[T]` -type is an instance of it: - -.. code-block:: nim - :test: "nim c $1" - - import sugar, typetraits - - type - Functor[A] = concept f - type MatchedGenericType = genericHead(typeof(f)) - # `f` will be a value of a type such as `Option[T]` - # `MatchedGenericType` will become the `Option` type - - f.val is A - # The Functor should provide a way to obtain - # a value stored inside it - - type T = auto - map(f, A -> T) is MatchedGenericType[T] - # And it should provide a way to map one instance of - # the Functor to a instance of a different type, given - # a suitable `map` operation for the enclosed values - - import options - echo Option[int] is Functor # prints true - - -Concept derived values ----------------------- - -All top level constants or types appearing within the concept body are -accessible through the dot operator in procs where the concept was successfully -matched to a concrete type: - -.. code-block:: nim - type - DateTime = concept t1, t2, type T - const Min = T.MinDate - T.Now is T - - t1 < t2 is bool - - type TimeSpan = typeof(t1 - t2) - TimeSpan * int is TimeSpan - TimeSpan + TimeSpan is TimeSpan - - t1 + TimeSpan is T - - proc eventsJitter(events: Enumerable[DateTime]): float = - var - # this variable will have the inferred TimeSpan type for - # the concrete Date-like value the proc was called with: - averageInterval: DateTime.TimeSpan - - deviation: float - ... - - -Concept refinement ------------------- - -When the matched type within a concept is directly tested against a different -concept, we say that the outer concept is a refinement of the inner concept and -thus it is more-specific. When both concepts are matched in a call during -overload resolution, Nim will assign a higher precedence to the most specific -one. As an alternative way of defining concept refinements, you can use the -object inheritance syntax involving the ``of`` keyword: - -.. code-block:: nim - type - Graph = concept g, type G of EquallyComparable, Copyable - type - VertexType = G.VertexType - EdgeType = G.EdgeType - - VertexType is Copyable - EdgeType is Copyable - - var - v: VertexType - e: EdgeType - - IncidendeGraph = concept of Graph - # symbols such as variables and types from the refined - # concept are automatically in scope: - - g.source(e) is VertexType - g.target(e) is VertexType - - g.outgoingEdges(v) is Enumerable[EdgeType] - - BidirectionalGraph = concept g, type G - # The following will also turn the concept into a refinement when it - # comes to overload resolution, but it doesn't provide the convenient - # symbol inheritance - g is IncidendeGraph - - g.incomingEdges(G.VertexType) is Enumerable[G.EdgeType] - - proc f(g: IncidendeGraph) - proc f(g: BidirectionalGraph) # this one will be preferred if we pass a type - # matching the BidirectionalGraph concept - -.. - Converter type classes - ---------------------- - - Concepts can also be used to convert a whole range of types to a single type or - a small set of simpler types. This is achieved with a `return` statement within - the concept body: - - .. code-block:: nim - type - Stringable = concept x - $x is string - return $x - - StringRefValue[CharType] = object - base: ptr CharType - len: int - - StringRef = concept x - # the following would be an overloaded proc for cstring, string, seq and - # other user-defined types, returning either a StringRefValue[char] or - # StringRefValue[wchar] - return makeStringRefValue(x) - - # the varargs param will here be converted to an array of StringRefValues - # the proc will have only two instantiations for the two character types - proc log(format: static string, varargs[StringRef]) - - # this proc will allow char and wchar values to be mixed in - # the same call at the cost of additional instantiations - # the varargs param will be converted to a tuple - proc log(format: static string, varargs[distinct StringRef]) - - -.. - VTable types - ------------ - - Concepts allow Nim to define a great number of algorithms, using only - static polymorphism and without erasing any type information or sacrificing - any execution speed. But when polymorphic collections of objects are required, - the user must use one of the provided type erasure techniques - either common - base types or VTable types. - - VTable types are represented as "fat pointers" storing a reference to an - object together with a reference to a table of procs implementing a set of - required operations (the so called vtable). - - In contrast to other programming languages, the vtable in Nim is stored - externally to the object, allowing you to create multiple different vtable - views for the same object. Thus, the polymorphism in Nim is unbounded - - any type can implement an unlimited number of protocols or interfaces not - originally envisioned by the type's author. - - Any concept type can be turned into a VTable type by using the ``vtref`` - or the ``vtptr`` compiler magics. Under the hood, these magics generate - a converter type class, which converts the regular instances of the matching - types to the corresponding VTable type. - - .. code-block:: nim - type - IntEnumerable = vtref Enumerable[int] - - MyObject = object - enumerables: seq[IntEnumerable] - streams: seq[OutputStream.vtref] - - proc addEnumerable(o: var MyObject, e: IntEnumerable) = - o.enumerables.add e - - proc addStream(o: var MyObject, e: OutputStream.vtref) = - o.streams.add e - - The procs that will be included in the vtable are derived from the concept - body and include all proc calls for which all param types were specified as - concrete types. All such calls should include exactly one param of the type - matched against the concept (not necessarily in the first position), which - will be considered the value bound to the vtable. - - Overloads will be created for all captured procs, accepting the vtable type - in the position of the captured underlying object. - - Under these rules, it's possible to obtain a vtable type for a concept with - unbound type parameters or one instantiated with metatypes (type classes), - but it will include a smaller number of captured procs. A completely empty - vtable will be reported as an error. - - The ``vtref`` magic produces types which can be bound to ``ref`` types and - the ``vtptr`` magic produced types bound to ``ptr`` types. - - -Type bound operations -===================== - -There are 4 operations that are bound to a type: - -1. Assignment -2. Moves -3. Destruction -4. Deep copying for communication between threads - -These operations can be *overridden* instead of *overloaded*. This means the -implementation is automatically lifted to structured types. For instance if type -``T`` has an overridden assignment operator ``=`` this operator is also used -for assignments of the type ``seq[T]``. Since these operations are bound to a -type they have to be bound to a nominal type for reasons of simplicity of -implementation: This means an overridden ``deepCopy`` for ``ref T`` is really -bound to ``T`` and not to ``ref T``. This also means that one cannot override -``deepCopy`` for both ``ptr T`` and ``ref T`` at the same time; instead a -helper distinct or object type has to be used for one pointer type. - -Assignments, moves and destruction are specified in -the `destructors <destructors.html>`_ document. - - -deepCopy --------- - -``=deepCopy`` is a builtin that is invoked whenever data is passed to -a ``spawn``'ed proc to ensure memory safety. The programmer can override its -behaviour for a specific ``ref`` or ``ptr`` type ``T``. (Later versions of the -language may weaken this restriction.) - -The signature has to be: - -.. code-block:: nim - proc `=deepCopy`(x: T): T - -This mechanism will be used by most data structures that support shared memory -like channels to implement thread safe automatic memory management. - -The builtin ``deepCopy`` can even clone closures and their environments. See -the documentation of `spawn <#parallel-amp-spawn-spawn-statement>`_ for details. - - -Case statement macros -===================== - -A macro that needs to be called `match`:idx: can be used to rewrite -``case`` statements in order to implement `pattern matching`:idx: for -certain types. The following example implements a simplistic form of -pattern matching for tuples, leveraging the existing equality operator -for tuples (as provided in ``system.==``): - -.. code-block:: nim - :test: "nim c $1" - - {.experimental: "caseStmtMacros".} - - import macros - - macro match(n: tuple): untyped = - result = newTree(nnkIfStmt) - let selector = n[0] - for i in 1 ..< n.len: - let it = n[i] - case it.kind - of nnkElse, nnkElifBranch, nnkElifExpr, nnkElseExpr: - result.add it - of nnkOfBranch: - for j in 0..it.len-2: - let cond = newCall("==", selector, it[j]) - result.add newTree(nnkElifBranch, cond, it[^1]) - else: - error "'match' cannot handle this node", it - echo repr result - - case ("foo", 78) - of ("foo", 78): echo "yes" - of ("bar", 88): echo "no" - else: discard - - -Currently case statement macros must be enabled explicitly -via ``{.experimental: "caseStmtMacros".}``. - -``match`` macros are subject to overload resolution. First the -``case``'s selector expression is used to determine which ``match`` -macro to call. To this macro is then passed the complete ``case`` -statement body and the macro is evaluated. - -In other words, the macro needs to transform the full ``case`` statement -but only the statement's selector expression is used to determine which -macro to call. - - -For loop macros ---------------- - -A macro that takes as its only input parameter an expression of the special -type ``system.ForLoopStmt`` can rewrite the entirety of a ``for`` loop: - -.. code-block:: nim - :test: "nim c $1" - - import macros - {.experimental: "forLoopMacros".} - - macro enumerate(x: ForLoopStmt): untyped = - expectKind x, nnkForStmt - # we strip off the first for loop variable and use - # it as an integer counter: - result = newStmtList() - result.add newVarStmt(x[0], newLit(0)) - var body = x[^1] - if body.kind != nnkStmtList: - body = newTree(nnkStmtList, body) - body.add newCall(bindSym"inc", x[0]) - var newFor = newTree(nnkForStmt) - for i in 1..x.len-3: - newFor.add x[i] - # transform enumerate(X) to 'X' - newFor.add x[^2][1] - newFor.add body - result.add newFor - # now wrap the whole macro in a block to create a new scope - result = quote do: - block: `result` - - for a, b in enumerate(items([1, 2, 3])): - echo a, " ", b - - # without wrapping the macro in a block, we'd need to choose different - # names for `a` and `b` here to avoid redefinition errors - for a, b in enumerate([1, 2, 3, 5]): - echo a, " ", b - - -Currently for loop macros must be enabled explicitly -via ``{.experimental: "forLoopMacros".}``. - - -Term rewriting macros -===================== - -Term rewriting macros are macros or templates that have not only -a *name* but also a *pattern* that is searched for after the semantic checking -phase of the compiler: This means they provide an easy way to enhance the -compilation pipeline with user defined optimizations: - -.. code-block:: nim - template optMul{`*`(a, 2)}(a: int): int = a+a - - let x = 3 - echo x * 2 - -The compiler now rewrites ``x * 2`` as ``x + x``. The code inside the -curlies is the pattern to match against. The operators ``*``, ``**``, -``|``, ``~`` have a special meaning in patterns if they are written in infix -notation, so to match verbatim against ``*`` the ordinary function call syntax -needs to be used. - -Term rewriting macro are applied recursively, up to a limit. This means that -if the result of a term rewriting macro is eligible for another rewriting, -the compiler will try to perform it, and so on, until no more optimizations -are applicable. To avoid putting the compiler into an infinite loop, there is -a hard limit on how many times a single term rewriting macro can be applied. -Once this limit has been passed, the term rewriting macro will be ignored. - -Unfortunately optimizations are hard to get right and even the tiny example -is **wrong**: - -.. code-block:: nim - template optMul{`*`(a, 2)}(a: int): int = a+a - - proc f(): int = - echo "side effect!" - result = 55 - - echo f() * 2 - -We cannot duplicate 'a' if it denotes an expression that has a side effect! -Fortunately Nim supports side effect analysis: - -.. code-block:: nim - template optMul{`*`(a, 2)}(a: int{noSideEffect}): int = a+a - - proc f(): int = - echo "side effect!" - result = 55 - - echo f() * 2 # not optimized ;-) - -You can make one overload matching with a constraint and one without, and the -one with a constraint will have precedence, and so you can handle both cases -differently. - -So what about ``2 * a``? We should tell the compiler ``*`` is commutative. We -cannot really do that however as the following code only swaps arguments -blindly: - -.. code-block:: nim - template mulIsCommutative{`*`(a, b)}(a, b: int): int = b*a - -What optimizers really need to do is a *canonicalization*: - -.. code-block:: nim - template canonMul{`*`(a, b)}(a: int{lit}, b: int): int = b*a - -The ``int{lit}`` parameter pattern matches against an expression of -type ``int``, but only if it's a literal. - - - -Parameter constraints ---------------------- - -The `parameter constraint`:idx: expression can use the operators ``|`` (or), -``&`` (and) and ``~`` (not) and the following predicates: - -=================== ===================================================== -Predicate Meaning -=================== ===================================================== -``atom`` The matching node has no children. -``lit`` The matching node is a literal like "abc", 12. -``sym`` The matching node must be a symbol (a bound - identifier). -``ident`` The matching node must be an identifier (an unbound - identifier). -``call`` The matching AST must be a call/apply expression. -``lvalue`` The matching AST must be an lvalue. -``sideeffect`` The matching AST must have a side effect. -``nosideeffect`` The matching AST must have no side effect. -``param`` A symbol which is a parameter. -``genericparam`` A symbol which is a generic parameter. -``module`` A symbol which is a module. -``type`` A symbol which is a type. -``var`` A symbol which is a variable. -``let`` A symbol which is a ``let`` variable. -``const`` A symbol which is a constant. -``result`` The special ``result`` variable. -``proc`` A symbol which is a proc. -``method`` A symbol which is a method. -``iterator`` A symbol which is an iterator. -``converter`` A symbol which is a converter. -``macro`` A symbol which is a macro. -``template`` A symbol which is a template. -``field`` A symbol which is a field in a tuple or an object. -``enumfield`` A symbol which is a field in an enumeration. -``forvar`` A for loop variable. -``label`` A label (used in ``block`` statements). -``nk*`` The matching AST must have the specified kind. - (Example: ``nkIfStmt`` denotes an ``if`` statement.) -``alias`` States that the marked parameter needs to alias - with *some* other parameter. -``noalias`` States that *every* other parameter must not alias - with the marked parameter. -=================== ===================================================== - -Predicates that share their name with a keyword have to be escaped with -backticks. -The ``alias`` and ``noalias`` predicates refer not only to the matching AST, -but also to every other bound parameter; syntactically they need to occur after -the ordinary AST predicates: - -.. code-block:: nim - template ex{a = b + c}(a: int{noalias}, b, c: int) = - # this transformation is only valid if 'b' and 'c' do not alias 'a': - a = b - inc a, c - - -Pattern operators ------------------ - -The operators ``*``, ``**``, ``|``, ``~`` have a special meaning in patterns -if they are written in infix notation. - - -The ``|`` operator -~~~~~~~~~~~~~~~~~~ - -The ``|`` operator if used as infix operator creates an ordered choice: - -.. code-block:: nim - template t{0|1}(): untyped = 3 - let a = 1 - # outputs 3: - echo a - -The matching is performed after the compiler performed some optimizations like -constant folding, so the following does not work: - -.. code-block:: nim - template t{0|1}(): untyped = 3 - # outputs 1: - echo 1 - -The reason is that the compiler already transformed the 1 into "1" for -the ``echo`` statement. However, a term rewriting macro should not change the -semantics anyway. In fact they can be deactivated with the ``--patterns:off`` -command line option or temporarily with the ``patterns`` pragma. - - -The ``{}`` operator -~~~~~~~~~~~~~~~~~~~ - -A pattern expression can be bound to a pattern parameter via the ``expr{param}`` -notation: - -.. code-block:: nim - template t{(0|1|2){x}}(x: untyped): untyped = x+1 - let a = 1 - # outputs 2: - echo a - - -The ``~`` operator -~~~~~~~~~~~~~~~~~~ - -The ``~`` operator is the **not** operator in patterns: - -.. code-block:: nim - template t{x = (~x){y} and (~x){z}}(x, y, z: bool) = - x = y - if x: x = z - - var - a = false - b = true - c = false - a = b and c - echo a - - -The ``*`` operator -~~~~~~~~~~~~~~~~~~ - -The ``*`` operator can *flatten* a nested binary expression like ``a & b & c`` -to ``&(a, b, c)``: - -.. code-block:: nim - var - calls = 0 - - proc `&&`(s: varargs[string]): string = - result = s[0] - for i in 1..len(s)-1: result.add s[i] - inc calls - - template optConc{ `&&` * a }(a: string): untyped = &&a - - let space = " " - echo "my" && (space & "awe" && "some " ) && "concat" - - # check that it's been optimized properly: - doAssert calls == 1 - - -The second operator of `*` must be a parameter; it is used to gather all the -arguments. The expression ``"my" && (space & "awe" && "some " ) && "concat"`` -is passed to ``optConc`` in ``a`` as a special list (of kind ``nkArgList``) -which is flattened into a call expression; thus the invocation of ``optConc`` -produces: - -.. code-block:: nim - `&&`("my", space & "awe", "some ", "concat") - - -The ``**`` operator -~~~~~~~~~~~~~~~~~~~ - -The ``**`` is much like the ``*`` operator, except that it gathers not only -all the arguments, but also the matched operators in reverse polish notation: - -.. code-block:: nim - import macros - - type - Matrix = object - dummy: int - - proc `*`(a, b: Matrix): Matrix = discard - proc `+`(a, b: Matrix): Matrix = discard - proc `-`(a, b: Matrix): Matrix = discard - proc `$`(a: Matrix): string = result = $a.dummy - proc mat21(): Matrix = - result.dummy = 21 - - macro optM{ (`+`|`-`|`*`) ** a }(a: Matrix): untyped = - echo treeRepr(a) - result = newCall(bindSym"mat21") - - var x, y, z: Matrix - - echo x + y * z - x - -This passes the expression ``x + y * z - x`` to the ``optM`` macro as -an ``nnkArgList`` node containing:: - - Arglist - Sym "x" - Sym "y" - Sym "z" - Sym "*" - Sym "+" - Sym "x" - Sym "-" - -(Which is the reverse polish notation of ``x + y * z - x``.) - - -Parameters ----------- - -Parameters in a pattern are type checked in the matching process. If a -parameter is of the type ``varargs`` it is treated specially and it can match -0 or more arguments in the AST to be matched against: - -.. code-block:: nim - template optWrite{ - write(f, x) - ((write|writeLine){w})(f, y) - }(x, y: varargs[untyped], f: File, w: untyped) = - w(f, x, y) - - - -Example: Partial evaluation ---------------------------- - -The following example shows how some simple partial evaluation can be -implemented with term rewriting: - -.. code-block:: nim - proc p(x, y: int; cond: bool): int = - result = if cond: x + y else: x - y - - template optP1{p(x, y, true)}(x, y: untyped): untyped = x + y - template optP2{p(x, y, false)}(x, y: untyped): untyped = x - y - - -Example: Hoisting ------------------ - -The following example shows how some form of hoisting can be implemented: - -.. code-block:: nim - import pegs - - template optPeg{peg(pattern)}(pattern: string{lit}): Peg = - var gl {.global, gensym.} = peg(pattern) - gl - - for i in 0 .. 3: - echo match("(a b c)", peg"'(' @ ')'") - echo match("W_HI_Le", peg"\y 'while'") - -The ``optPeg`` template optimizes the case of a peg constructor with a string -literal, so that the pattern will only be parsed once at program startup and -stored in a global ``gl`` which is then re-used. This optimization is called -hoisting because it is comparable to classical loop hoisting. - - -AST based overloading -===================== - -Parameter constraints can also be used for ordinary routine parameters; these -constraints affect ordinary overloading resolution then: - -.. code-block:: nim - proc optLit(a: string{lit|`const`}) = - echo "string literal" - proc optLit(a: string) = - echo "no string literal" - - const - constant = "abc" - - var - variable = "xyz" - - optLit("literal") - optLit(constant) - optLit(variable) - -However, the constraints ``alias`` and ``noalias`` are not available in -ordinary routines. - - -Parallel & Spawn -================ - -Nim has two flavors of parallelism: -1) `Structured`:idx: parallelism via the ``parallel`` statement. -2) `Unstructured`:idx: parallelism via the standalone ``spawn`` statement. - -Nim has a builtin thread pool that can be used for CPU intensive tasks. For -IO intensive tasks the ``async`` and ``await`` features should be -used instead. Both parallel and spawn need the `threadpool <threadpool.html>`_ -module to work. - -Somewhat confusingly, ``spawn`` is also used in the ``parallel`` statement -with slightly different semantics. ``spawn`` always takes a call expression of -the form ``f(a, ...)``. Let ``T`` be ``f``'s return type. If ``T`` is ``void`` -then ``spawn``'s return type is also ``void`` otherwise it is ``FlowVar[T]``. - -Within a ``parallel`` section sometimes the ``FlowVar[T]`` is eliminated -to ``T``. This happens when ``T`` does not contain any GC'ed memory. -The compiler can ensure the location in ``location = spawn f(...)`` is not -read prematurely within a ``parallel`` section and so there is no need for -the overhead of an indirection via ``FlowVar[T]`` to ensure correctness. - -**Note**: Currently exceptions are not propagated between ``spawn``'ed tasks! - - -Spawn statement ---------------- - -`spawn`:idx: can be used to pass a task to the thread pool: - -.. code-block:: nim - import threadpool - - proc processLine(line: string) = - discard "do some heavy lifting here" - - for x in lines("myinput.txt"): - spawn processLine(x) - sync() - -For reasons of type safety and implementation simplicity the expression -that ``spawn`` takes is restricted: - -* It must be a call expression ``f(a, ...)``. -* ``f`` must be ``gcsafe``. -* ``f`` must not have the calling convention ``closure``. -* ``f``'s parameters may not be of type ``var``. - This means one has to use raw ``ptr``'s for data passing reminding the - programmer to be careful. -* ``ref`` parameters are deeply copied which is a subtle semantic change and - can cause performance problems but ensures memory safety. This deep copy - is performed via ``system.deepCopy`` and so can be overridden. -* For *safe* data exchange between ``f`` and the caller a global ``TChannel`` - needs to be used. However, since spawn can return a result, often no further - communication is required. - - -``spawn`` executes the passed expression on the thread pool and returns -a `data flow variable`:idx: ``FlowVar[T]`` that can be read from. The reading -with the ``^`` operator is **blocking**. However, one can use ``blockUntilAny`` to -wait on multiple flow variables at the same time: - -.. code-block:: nim - import threadpool, ... - - # wait until 2 out of 3 servers received the update: - proc main = - var responses = newSeq[FlowVarBase](3) - for i in 0..2: - responses[i] = spawn tellServer(Update, "key", "value") - var index = blockUntilAny(responses) - assert index >= 0 - responses.del(index) - discard blockUntilAny(responses) - -Data flow variables ensure that no data races -are possible. Due to technical limitations not every type ``T`` is possible in -a data flow variable: ``T`` has to be of the type ``ref``, ``string``, ``seq`` -or of a type that doesn't contain a type that is garbage collected. This -restriction is not hard to work-around in practice. - - - -Parallel statement ------------------- - -Example: - -.. code-block:: nim - :test: "nim c --threads:on $1" - - # Compute PI in an inefficient way - import strutils, math, threadpool - {.experimental: "parallel".} - - proc term(k: float): float = 4 * math.pow(-1, k) / (2*k + 1) - - proc pi(n: int): float = - var ch = newSeq[float](n+1) - parallel: - for k in 0..ch.high: - ch[k] = spawn term(float(k)) - for k in 0..ch.high: - result += ch[k] - - echo formatFloat(pi(5000)) - - -The parallel statement is the preferred mechanism to introduce parallelism in a -Nim program. A subset of the Nim language is valid within a ``parallel`` -section. This subset is checked during semantic analysis to be free of data -races. A sophisticated `disjoint checker`:idx: ensures that no data races are -possible even though shared memory is extensively supported! - -The subset is in fact the full language with the following -restrictions / changes: - -* ``spawn`` within a ``parallel`` section has special semantics. -* Every location of the form ``a[i]`` and ``a[i..j]`` and ``dest`` where - ``dest`` is part of the pattern ``dest = spawn f(...)`` has to be - provably disjoint. This is called the *disjoint check*. -* Every other complex location ``loc`` that is used in a spawned - proc (``spawn f(loc)``) has to be immutable for the duration of - the ``parallel`` section. This is called the *immutability check*. Currently - it is not specified what exactly "complex location" means. We need to make - this an optimization! -* Every array access has to be provably within bounds. This is called - the *bounds check*. -* Slices are optimized so that no copy is performed. This optimization is not - yet performed for ordinary slices outside of a ``parallel`` section. - - -Guards and locks -================ - -Apart from ``spawn`` and ``parallel`` Nim also provides all the common low level -concurrency mechanisms like locks, atomic intrinsics or condition variables. - -Nim significantly improves on the safety of these features via additional -pragmas: - -1) A `guard`:idx: annotation is introduced to prevent data races. -2) Every access of a guarded memory location needs to happen in an - appropriate `locks`:idx: statement. -3) Locks and routines can be annotated with `lock levels`:idx: to allow - potential deadlocks to be detected during semantic analysis. - - -Guards and the locks section ----------------------------- - -Protecting global variables -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Object fields and global variables can be annotated via a ``guard`` pragma: - -.. code-block:: nim - var glock: TLock - var gdata {.guard: glock.}: int - -The compiler then ensures that every access of ``gdata`` is within a ``locks`` -section: - -.. code-block:: nim - proc invalid = - # invalid: unguarded access: - echo gdata - - proc valid = - # valid access: - {.locks: [glock].}: - echo gdata - -Top level accesses to ``gdata`` are always allowed so that it can be initialized -conveniently. It is *assumed* (but not enforced) that every top level statement -is executed before any concurrent action happens. - -The ``locks`` section deliberately looks ugly because it has no runtime -semantics and should not be used directly! It should only be used in templates -that also implement some form of locking at runtime: - -.. code-block:: nim - template lock(a: TLock; body: untyped) = - pthread_mutex_lock(a) - {.locks: [a].}: - try: - body - finally: - pthread_mutex_unlock(a) - - -The guard does not need to be of any particular type. It is flexible enough to -model low level lockfree mechanisms: - -.. code-block:: nim - var dummyLock {.compileTime.}: int - var atomicCounter {.guard: dummyLock.}: int - - template atomicRead(x): untyped = - {.locks: [dummyLock].}: - memoryReadBarrier() - x - - echo atomicRead(atomicCounter) - - -The ``locks`` pragma takes a list of lock expressions ``locks: [a, b, ...]`` -in order to support *multi lock* statements. Why these are essential is -explained in the `lock levels <#guards-and-locks-lock-levels>`_ section. - - -Protecting general locations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``guard`` annotation can also be used to protect fields within an object. -The guard then needs to be another field within the same object or a -global variable. - -Since objects can reside on the heap or on the stack this greatly enhances the -expressivity of the language: - -.. code-block:: nim - type - ProtectedCounter = object - v {.guard: L.}: int - L: TLock - - proc incCounters(counters: var openArray[ProtectedCounter]) = - for i in 0..counters.high: - lock counters[i].L: - inc counters[i].v - -The access to field ``x.v`` is allowed since its guard ``x.L`` is active. -After template expansion, this amounts to: - -.. code-block:: nim - proc incCounters(counters: var openArray[ProtectedCounter]) = - for i in 0..counters.high: - pthread_mutex_lock(counters[i].L) - {.locks: [counters[i].L].}: - try: - inc counters[i].v - finally: - pthread_mutex_unlock(counters[i].L) - -There is an analysis that checks that ``counters[i].L`` is the lock that -corresponds to the protected location ``counters[i].v``. This analysis is called -`path analysis`:idx: because it deals with paths to locations -like ``obj.field[i].fieldB[j]``. - -The path analysis is **currently unsound**, but that doesn't make it useless. -Two paths are considered equivalent if they are syntactically the same. - -This means the following compiles (for now) even though it really should not: - -.. code-block:: nim - {.locks: [a[i].L].}: - inc i - access a[i].v - - - -Lock levels ------------ - -Lock levels are used to enforce a global locking order in order to detect -potential deadlocks during semantic analysis. A lock level is an constant -integer in the range 0..1_000. Lock level 0 means that no lock is acquired at -all. - -If a section of code holds a lock of level ``M`` than it can also acquire any -lock of level ``N < M``. Another lock of level ``M`` cannot be acquired. Locks -of the same level can only be acquired *at the same time* within a -single ``locks`` section: - -.. code-block:: nim - var a, b: TLock[2] - var x: TLock[1] - # invalid locking order: TLock[1] cannot be acquired before TLock[2]: - {.locks: [x].}: - {.locks: [a].}: - ... - # valid locking order: TLock[2] acquired before TLock[1]: - {.locks: [a].}: - {.locks: [x].}: - ... - - # invalid locking order: TLock[2] acquired before TLock[2]: - {.locks: [a].}: - {.locks: [b].}: - ... - - # valid locking order, locks of the same level acquired at the same time: - {.locks: [a, b].}: - ... - - -Here is how a typical multilock statement can be implemented in Nim. Note how -the runtime check is required to ensure a global ordering for two locks ``a`` -and ``b`` of the same lock level: - -.. code-block:: nim - template multilock(a, b: ptr TLock; body: untyped) = - if cast[ByteAddress](a) < cast[ByteAddress](b): - pthread_mutex_lock(a) - pthread_mutex_lock(b) - else: - pthread_mutex_lock(b) - pthread_mutex_lock(a) - {.locks: [a, b].}: - try: - body - finally: - pthread_mutex_unlock(a) - pthread_mutex_unlock(b) - - -Whole routines can also be annotated with a ``locks`` pragma that takes a lock -level. This then means that the routine may acquire locks of up to this level. -This is essential so that procs can be called within a ``locks`` section: - -.. code-block:: nim - proc p() {.locks: 3.} = discard - - var a: TLock[4] - {.locks: [a].}: - # p's locklevel (3) is strictly less than a's (4) so the call is allowed: - p() - - -As usual ``locks`` is an inferred effect and there is a subtype -relation: ``proc () {.locks: N.}`` is a subtype of ``proc () {.locks: M.}`` -iff (M <= N). - -The ``locks`` pragma can also take the special value ``"unknown"``. This -is useful in the context of dynamic method dispatching. In the following -example, the compiler can infer a lock level of 0 for the ``base`` case. -However, one of the overloaded methods calls a procvar which is -potentially locking. Thus, the lock level of calling ``g.testMethod`` -cannot be inferred statically, leading to compiler warnings. By using -``{.locks: "unknown".}``, the base method can be marked explicitly as -having unknown lock level as well: - -.. code-block:: nim - type SomeBase* = ref object of RootObj - type SomeDerived* = ref object of SomeBase - memberProc*: proc () - - method testMethod(g: SomeBase) {.base, locks: "unknown".} = discard - method testMethod(g: SomeDerived) = - if g.memberProc != nil: - g.memberProc() - - -noRewrite pragma ----------------- - -Term rewriting macros and templates are currently greedy and -they will rewrite as long as there is a match. -There was no way to ensure some rewrite happens only once, -eg. when rewriting term to same term plus extra content. - -``noRewrite`` pragma can actually prevent further rewriting on marked code, -e.g. with given example ``echo("ab")`` will be rewritten just once: - -.. code-block:: nim - template pwnEcho{echo(x)}(x: expr) = - {.noRewrite.}: echo("pwned!") - - echo "ab" - -``noRewrite`` pragma can be useful to control term-rewriting macros recursion. - - -Taint mode -========== - -The Nim compiler and most parts of the standard library support -a taint mode. Input strings are declared with the `TaintedString`:idx: -string type declared in the ``system`` module. - -If the taint mode is turned on (via the ``--taintMode:on`` command line -option) it is a distinct string type which helps to detect input -validation errors: - -.. code-block:: nim - echo "your name: " - var name: TaintedString = stdin.readline - # it is safe here to output the name without any input validation, so - # we simply convert `name` to string to make the compiler happy: - echo "hi, ", name.string - -If the taint mode is turned off, ``TaintedString`` is simply an alias for -``string``. - - -Aliasing restrictions in parameter passing -========================================== - -**Note**: The aliasing restrictions are currently not enforced by the -implementation and need to be fleshed out further. - -"Aliasing" here means that the underlying storage locations overlap in memory -at runtime. An "output parameter" is a parameter of type ``var T``, -an input parameter is any parameter that is not of type ``var``. - -1. Two output parameters should never be aliased. -2. An input and an output parameter should not be aliased. -3. An output parameter should never be aliased with a global or thread local - variable referenced by the called proc. -4. An input parameter should not be aliased with a global or thread local - variable updated by the called proc. - -One problem with rules 3 and 4 is that they affect specific global or thread -local variables, but Nim's effect tracking only tracks "uses no global variable" -via ``.noSideEffect``. The rules 3 and 4 can also be approximated by a different rule: - -5. A global or thread local variable (or a location derived from such a location) - can only passed to a parameter of a ``.noSideEffect`` proc. diff --git a/doc/manual_experimental_strictnotnil.md b/doc/manual_experimental_strictnotnil.md new file mode 100644 index 000000000..a6fa6cda8 --- /dev/null +++ b/doc/manual_experimental_strictnotnil.md @@ -0,0 +1,255 @@ +Strict not nil checking +========================= + +.. default-role:: code +.. include:: rstcommon.rst + +**Note:** This feature is experimental, you need to enable it with + + ```nim + {.experimental: "strictNotNil".} + ``` + +or + + ```cmd + nim c --experimental:strictNotNil <program> + ``` + +In the second case it would check builtin and imported modules as well. + +It checks the nilability of ref-like types and makes dereferencing safer based on flow typing and `not nil` annotations. + +Its implementation is different than the `notnil` one: defined under `strictNotNil`. Keep in mind the difference in option names, be careful with distinguishing them. + +We check several kinds of types for nilability: + +- ref types +- pointer types +- proc types +- cstrings + +nil +------- + +The default kind of nilability types is the nilable kind: they can have the value `nil`. +If you have a non-nilable type `T`, you can use `T nil` to get a nilable type for it. + + +not nil +-------- + +You can annotate a type where nil isn't a valid value with `not nil`. + + ```nim + type + NilableObject = ref object + a: int + Object = NilableObject not nil + + Proc = (proc (x, y: int)) + + proc p(x: Object) = + echo x.a # ensured to dereference without an error + # compiler catches this: + p(nil) + # and also this: + var x: NilableObject + if x.isNil: + p(x) + else: + p(x) # ok + ``` + + + +If a type can include `nil` as a valid value, dereferencing values of the type +is checked by the compiler: if a value which might be nil is dereferenced, this +produces a warning by default, you can turn this into an error using +the compiler options `--warningAsError:strictNotNil`:option:. + +If a type is nilable, you should dereference its values only after a `isNil` or equivalent check. + +local turn on/off +--------------------- + +You can still turn off nil checking on function/module level by using a `{.strictNotNil: off.}` pragma. + +.. + Note: test that/TODO for code/manual. + +nilability state +----------------- + +Currently, a nilable value can be `Safe`, `MaybeNil` or `Nil` : we use internally `Parent` and `Unreachable` but this is an implementation detail(a parent layer has the actual nilability). + +- `Safe` means it shouldn't be nil at that point: e.g. after assignment to + a non-nil value or `not a.isNil` check +- `MaybeNil` means it might be nil, but it might not be nil: e.g. an argument, + a call argument or a value after an `if` and `else`. +- `Nil` means it should be nil at that point; e.g. after an assignment to + `nil` or a `.isNil` check. +- `Unreachable` means it shouldn't be possible to access this in this branch: + so we do generate a warning as well. + +We show an error for each dereference (`[]`, `.field`, `[index]` `()` etc.) which is of a tracked expression which is +in `MaybeNil` or `Nil` state. + + +type nilability +---------------- + +Types are either nilable or non-nilable. +When you pass a param or a default value, we use the type : for nilable types we return `MaybeNil` +and for non-nilable `Safe`. + +.. + TODO: fix the manual here. (This is not great, as default values for non-nilables and nilables are usually actually `nil` , so we should think a bit more about this section.) + +params rules +------------ + +Param's nilability is detected based on type nilability. We use the type of the argument to detect the nilability. + + +assignment rules +----------------- + +Let's say we have `left = right`. + +When we assign, we pass the right's nilability to the left's expression. There should be special handling of aliasing and compound expressions which we specify in their sections. (Assignment is a possible alias `move` or `move out`). + +call args rules +----------------- + +When we call with arguments, we have two cases when we might change the nilability. + + ```nim + callByVar(a) + ``` + +Here `callByVar` can re-assign `a`, so this might change `a`'s nilability, so we change it to `MaybeNil`. +This is also a possible aliasing `move out` (moving out of a current alias set). + + ```nim + call(a) + ``` + +Here `call` can change a field or element of `a`, so if we have a dependant expression of `a` : e.g. `a.field`. Dependants become `MaybeNil`. + + +branches rules +--------------- + +Branches are the reason we do nil checking like this: with flow checking. +Sources of branching are `if`, `while`, `for`, `and`, `or`, `case`, `try` and combinations with `return`, `break`, `continue` and `raise` + +We create a new layer/"scope" for each branch where we map expressions to nilability. This happens when we "fork": usually on the beginning of a construct. +When branches "join" we usually unify their expression maps or/and nilabilities. + +Merging usually merges maps and alias sets: nilabilities are merged like this: + + ```nim + template union(l: Nilability, r: Nilability): Nilability = + ## unify two states + if l == r: + l + else: + MaybeNil + ``` + +Special handling is for `.isNil` and `== nil`, also for `not`, `and` and `or`. + +`not` reverses the nilability, `and` is similar to "forking" : the right expression is checked in the layer resulting from the left one and `or` is similar to "merging": the right and left expression should be both checked in the original layer. + +`isNil`, `== nil` make expressions `Nil`. If there is a `not` or `!= nil`, they make them `Safe`. +We also reverse the nilability in the opposite branch: e.g. `else`. + +compound expressions: field, index expressions +----------------------------------------------- + +We want to track also field(dot) and index(bracket) expressions. + +We track some of those compound expressions which might be nilable as dependants of their bases: `a.field` is changed if `a` is moved (re-assigned), +similarly `a[index]` is dependent on `a` and `a.field.field` on `a.field`. + +When we move the base, we update dependants to `MaybeNil`. Otherwise, we usually start with type nilability. + +When we call args, we update the nilability of their dependants to `MaybeNil` as the calls usually can change them. +We might need to check for `strictFuncs` pure funcs and not do that then. + +For field expressions `a.field`, we calculate an integer value based on a hash of the tree and just accept equivalent trees as equivalent expressions. + +For item expression `a[index]`, we also calculate an integer value based on a hash of the tree and accept equivalent trees as equivalent expressions: for static values only. +For now, we support only constant indices: we don't track expression with no-const indices. For those we just report a warning even if they are safe for now: one can use a local variable to workaround. For loops this might be annoying: so one should be able to turn off locally the warning using the `{.warning[StrictNotNil]:off.}`. + +For bracket expressions, in the future we might count `a[<any>]` as the same general expression. +This means we should the index but otherwise handle it the same for assign (maybe "aliasing" all the non-static elements) and differentiate only for static: e.g. `a[0]` and `a[1]`. + +element tracking +----------------- + +When we assign an object construction, we should track the fields as well: + + + ```nim + var a = Nilable(field: Nilable()) # a : Safe, a.field: Safe + ``` + +Usually we just track the result of an expression: probably this should apply for elements in other cases as well. +Also related to tracking initialization of expressions/fields. + +unstructured control flow rules +------------------------------- + +Unstructured control flow keywords as `return`, `break`, `continue`, `raise` mean that we jump from a branch out. +This means that if there is code after the finishing of the branch, it would be run if one hasn't hit the direct parent branch of those: so it is similar to an `else`. In those cases we should use the reverse nilabilities for the local to the condition expressions. E.g. + + ```nim + for a in c: + if not a.isNil: + b() + break + code # here a: Nil , because if not, we would have breaked + ``` + + +aliasing +------------ + +We support alias detection for local expressions. + +We track sets of aliased expressions. We start with all nilable local expressions in separate sets. +Assignments and other changes to nilability can move / move out expressions of sets. + +`move`: Moving `left` to `right` means we remove `left` from its current set and unify it with the `right`'s set. +This means it stops being aliased with its previous aliases. + + ```nim + var left = b + left = right # moving left to right + ``` + +`move out`: Moving out `left` might remove it from the current set and ensure that it's in its own set as a single element. +e.g. + + + ```nim + var left = b + left = nil # moving out + ``` + +.. + initialization of non nilable and nilable values + ------------------------------------------------- + + TODO + +warnings and errors +--------------------- + +We show an error for each dereference (`[]`, `.field`, `[index]` `()` etc.) which is of a tracked expression which is +in `MaybeNil` or `Nil` state. + +We might also show a history of the transitions and the reasons for them that might change the nilability of the expression. + diff --git a/doc/markdown_rst.md b/doc/markdown_rst.md new file mode 100644 index 000000000..c7977f75a --- /dev/null +++ b/doc/markdown_rst.md @@ -0,0 +1,349 @@ +========================================== +Nim-flavored Markdown and reStructuredText +========================================== + +:Author: Andrey Makarov +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +.. importdoc:: docgen.md + +Both `Markdown`:idx: (md) and `reStructuredText`:idx: (RST) are markup +languages whose goal is to typeset texts with complex structure, +formatting and references using simple plaintext representation. + +Command line usage +================== + +Usage (to convert Markdown into HTML): + + ```cmd + nim md2html markdown_rst.md + ``` + +Output: + + You're reading it! + +The `md2tex`:option: command is invoked identically to `md2html`:option:, +but outputs a ``.tex`` file instead of ``.html``. + +These tools embedded into Nim compiler; the compiler can output +the result to HTML [^html] or Latex [^latex]. + +[^html]: commands `nim doc`:cmd: for ``*.nim`` files and + `nim rst2html`:cmd: for ``*.rst`` files + +[^latex]: commands `nim doc2tex`:cmd: for ``*.nim`` and + `nim rst2tex`:cmd: for ``*.rst``. + +Full list of supported commands: + +=================== ====================== ============ ============== +command runs on... input format output format +=================== ====================== ============ ============== +`nim md2html`:cmd: standalone md files ``.md`` ``.html`` HTML +`nim md2tex`:cmd: same same ``.tex`` LaTeX +`nim rst2html`:cmd: standalone rst files ``.rst`` ``.html`` HTML +`nim rst2tex`:cmd: same same ``.tex`` LaTeX +`nim doc`:cmd: documentation comments ``.nim`` ``.html`` HTML +`nim doc2tex`:cmd: same same ``.tex`` LaTeX +`nim jsondoc`:cmd: same same ``.json`` JSON +=================== ====================== ============ ============== + + +Basic markup +============ + +If you are new to Markdown/RST please consider reading the following: + +1) [Markdown Basic Syntax] +2) a long specification of Markdown: [CommonMark Spec] +3) a short [quick introduction] to RST +4) an [RST reference]: a comprehensive cheatsheet for RST +5) a more formal 50-page [RST specification]. + +Features +-------- + +A large subset is implemented with some [limitations] and +[additional Nim-specific features]. + +Supported common RST/Markdown features: + +* body elements + + sections + + transitions + + paragraphs + + bullet lists using \+, \*, \- + + enumerated lists using arabic numerals or alphabet + characters: 1. ... 2. ... *or* a. ... b. ... *or* A. ... B. ... + + footnotes (including manually numbered, auto-numbered, auto-numbered + with label, and auto-symbol footnotes) and citations + + field lists + + option lists + + quoted literal blocks + + line blocks + + simple tables + + directives (see official documentation in [RST directives list]): + - ``image``, ``figure`` for including images and videos + - ``code`` + - ``contents`` (table of contents), ``container``, ``raw`` + - ``include`` + - admonitions: "attention", "caution", "danger", "error", "hint", + "important", "note", "tip", "warning", "admonition" + - substitution definitions: `replace` and `image` + + comments +* inline markup + + *emphasis*, **strong emphasis**, + ``inline literals``, hyperlink references (including embedded URI), + substitution references, standalone hyperlinks, + internal links (inline and outline) + + \`interpreted text\` with roles ``:literal:``, ``:strong:``, + ``emphasis``, ``:sub:``/``:subscript:``, ``:sup:``/``:superscript:`` + (see [RST roles list] for description). + + inline internal targets + +RST mode only features +---------------------- + ++ RST syntax for definition lists (that is additional indentation after + a definition line) ++ indented literal blocks starting from ``::`` + +Markdown-specific features +-------------------------- + +* Markdown tables +* Markdown code blocks. For them the same additional arguments as for RST + code blocks can be provided (e.g. `test` or `number-lines`) but with + a one-line syntax like this: + + ```nim test number-lines=10 + echo "ok" + ``` +* Markdown links ``[...](...)`` +* Pandoc syntax for automatic links ``[...]``, see [Referencing] for description +* Pandoc syntax for footnotes, including ``[^10]`` (manually numbered) + and ``[^someName]`` (auto-numbered with a label) +* Markdown literal blocks indented by 4 or more spaces +* Markdown headlines +* Markdown block quotes +* Markdown syntax for definition lists +* using ``1`` as auto-enumerator in enumerated lists like RST ``#`` + (auto-enumerator ``1`` can not be used with ``#`` in the same list) + +Additional Nim-specific features +-------------------------------- + +* referencing to definitions in external files, see + [Markup external referencing] section +* directives: ``code-block`` [^Sphinx], ``title``, + ``index`` [^Sphinx] +* predefined roles + - ``:nim:`` (default), ``:c:`` (C programming language), + ``:python:``, ``:yaml:``, ``:java:``, ``:cpp:`` (C++), ``:csharp`` (C#). + That is every language that [highlite](highlite.html) supports. + They turn on appropriate syntax highlighting in inline code. + + .. Note:: default role for Nim files is ``:nim:``, + for ``*.rst`` it's currently ``:literal:``. + + - generic command line highlighting roles: + - ``:cmd:`` for commands and common shells syntax + - ``:console:`` the same for interactive sessions + (commands should be prepended by ``$``) + - ``:program:`` for executable names [^Sphinx] + (one can just use ``:cmd:`` on single word) + - ``:option:`` for command line options [^Sphinx] + - ``:tok:``, a role for highlighting of programming language tokens +* ***triple emphasis*** (bold and italic) using \*\*\* +* ``:idx:`` role for \`interpreted text\` to include the link to this + text into an index (example: [Nim index]). +* double slash `//` in option lists serves as a prefix for any option that + starts from a word (without any leading symbols like `-`, `--`, `/`): + + //compile compile the project + //doc generate documentation + + Here the dummy `//` will disappear, while options `compile`:option: + and `doc`:option: will be left in the final document. +* emoji / smiley symbols + +[^Sphinx]: similar but different from the directives of + Python [Sphinx directives] and [Sphinx roles] extensions + +.. Note:: By default Nim has ``roSupportMarkdown`` and + ``roSupportRawDirective`` turned **on**. + +.. warning:: Using Nim-specific features can cause other Markdown and + RST implementations to fail on your document. + +Referencing +=========== + +To be able to copy and share links Nim generates anchors for all +main document elements: + +* headlines (including document title) +* footnotes +* explicitly set anchors: RST internal cross-references and + inline internal targets +* Nim symbols (external referencing), see [Nim DocGen Tools Guide] for details. + +But direct use of those anchors have 2 problems: + +1. the anchors are usually mangled (e.g. spaces substituted to minus + signs, etc). +2. manual usage of anchors is not checked, so it's easy to get broken + links inside your project if e.g. spelling has changed for a heading + or you use a wrong relative path to your document. + +That's why Nim implementation has syntax for using +*original* labels for referencing. +Such referencing can be either local/internal or external: + +* Local referencing (inside any given file) is defined by + RST standard or Pandoc Markdown User guide. +* External (cross-document) referencing is a Nim-specific feature, + though it's not really different from local referencing by its syntax. + +Markup local referencing +------------------------ + +There are 2 syntax option available for referencing to objects +inside any given file, e.g. for headlines: + + Markdown RST + + Some headline Some headline + ============= ============= + + Ref. [Some headline] Ref. `Some headline`_ + + +Markup external referencing +--------------------------- + +The syntax is the same as for local referencing, but the anchors are +saved in ``.idx`` files, so one needs to generate them beforehand, +and they should be loaded by an `.. importdoc::` directive. +E.g. if we want to reference section "Some headline" in ``file1.md`` +from ``file2.md``, then ``file2.md`` may look like: + +``` +.. importdoc:: file1.md + +Ref. [Some headline] +``` + +```cmd +nim md2html --index:only file1.md # creates ``htmldocs/file1.idx`` +nim md2html file2.md # creates ``htmldocs/file2.html`` +``` + +To allow cross-references between any files in any order (especially, if +circular references are present), it's strongly recommended +to make a run for creating all the indexes first: + +```cmd +nim md2html --index:only file1.md # creates ``htmldocs/file1.idx`` +nim md2html --index:only file2.md # creates ``htmldocs/file2.idx`` +nim md2html file1.md # creates ``htmldocs/file1.html`` +nim md2html file2.md # creates ``htmldocs/file2.html`` +``` + +and then one can freely reference any objects as if these 2 documents +are actually 1 file. + +Other +===== + +Idiosyncrasies +-------------- + +Currently we do **not** aim at 100% Markdown or RST compatibility in inline +markup recognition rules because that would provide very little user value. +This parser has 2 modes for inline markup: + +1) Markdown-like mode which is enabled by `roPreferMarkdown` option + (turned **on** by default). + + .. Note:: RST features like directives are still turned **on** + +2) Compatibility mode which is RST rules. + +.. Note:: in both modes the parser interpretes text between single + backticks (code) identically: + backslash does not escape; the only exception: ``\`` folowed by ` + does escape so that we can always input a single backtick ` in + inline code. However that makes impossible to input code with + ``\`` at the end in *single* backticks, one must use *double* + backticks: + + `\` -- WRONG + ``\`` -- GOOD + So single backticks can always be input: `\`` will turn to ` code + +.. Attention:: + We don't support some obviously poor design choices of Markdown (or RST). + + - no support for the rule of 2 spaces causing a line break in Markdown + (use RST "line blocks" syntax for making line breaks) + + - interpretation of Markdown block quotes is also slightly different, + e.g. case + + >>> foo + > bar + >>baz + + is a single 3rd-level quote `foo bar baz` in original Markdown, while + in Nim we naturally see it as 3rd-level quote `foo` + 1st level `bar` + + 2nd level `baz`: + + >>> foo + > bar + >>baz + +Limitations +----------- + +* no Unicode support in character width calculations +* body elements + - no roman numerals in enumerated lists + - no doctest blocks + - no grid tables + - some directives are missing (check official [RST directives list]): + ``parsed-literal``, ``sidebar``, ``topic``, ``math``, ``rubric``, + ``epigraph``, ``highlights``, ``pull-quote``, ``compound``, + ``table``, ``csv-table``, ``list-table``, ``section-numbering``, + ``header``, ``footer``, ``meta``, ``class`` + - no ``role`` directives and no custom interpreted text roles + - some standard roles are not supported (check [RST roles list]) + - no generic admonition support +* inline markup + - no simple-inline-markup + - no embedded aliases + +Additional resources +-------------------- + +* See [Nim DocGen Tools Guide](docgen.html) for the details about + `nim doc`:cmd: command and idiosyncrasies of documentation markup in + ``.nim`` files and Nim programming language projects. +* See also documentation for [rst module](rst.html) -- Nim RST/Markdown parser. + +.. _Markdown Basic Syntax: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax +.. _CommonMark Spec: https://spec.commonmark.org/0.30 +.. _quick introduction: https://docutils.sourceforge.io/docs/user/rst/quickstart.html +.. _RST reference: https://docutils.sourceforge.io/docs/user/rst/quickref.html +.. _RST specification: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html +.. _RST directives list: https://docutils.sourceforge.io/docs/ref/rst/directives.html +.. _RST roles list: https://docutils.sourceforge.io/docs/ref/rst/roles.html +.. _Nim index: https://nim-lang.org/docs/theindex.html +.. _Sphinx directives: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html +.. _Sphinx roles: https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html diff --git a/doc/mm.md b/doc/mm.md new file mode 100644 index 000000000..5e0d2f3b9 --- /dev/null +++ b/doc/mm.md @@ -0,0 +1,95 @@ +======================= +Nim's Memory Management +======================= + +.. default-role:: code +.. include:: rstcommon.rst + +:Author: Andreas Rumpf +:Version: |nimversion| + +.. + + +> "The road to hell is paved with good intentions." + + +Multi-paradigm Memory Management Strategies +=========================================== + +.. default-role:: option + +Nim offers multiple different memory management strategies. +To choose the memory management strategy use the `--mm:` switch. + + .. hint:: **The recommended switch for newly written Nim code is `--mm:orc`.** + + +ARC/ORC +------- + +ORC is the default memory management strategy. It is a memory +management mode primarily based on reference counting. Reference cycles are +handled by a cycle collection mechanism based on "trial deletion". +Since algorithms based on "tracing" are not used, the runtime behavior is oblivious to the involved heap and stack sizes. + +The reference counting operations (= "RC ops") do not use atomic instructions and do not have to -- +instead entire subgraphs are *moved* between threads. The Nim compiler also aggressively +optimizes away RC ops and exploits [move semantics](destructors.html#move-semantics). + +Nim performs a fair share of optimizations for ARC/ORC; you can inspect what it did +to your time critical function via `--expandArc:functionName`. Likewise, you can inspect the whole module via `--expandArc:fileName`. + +`--mm:arc` uses the same mechanism as `--mm:orc`, but it leaves out the cycle collector. +Both ARC and ORC offer deterministic performance for `hard realtime`:idx: systems, but +ARC can be easier to reason about for people coming from Ada/C++/C -- roughly speaking +the memory for a variable is freed when it goes "out of scope". + +We generally advise you to use the `acyclic` annotation in order to optimize away the +cycle collector's overhead +but `--mm:orc` also produces more machine code than `--mm:arc`, so if you're on a target +where code size matters and you know that your code does not produce cycles, you can +use `--mm:arc`. Notice that the default `async`:idx: implementation produces cycles +and leaks memory with `--mm:arc`, in other words, for `async` you need to use `--mm:orc`. + + + +Other MM modes +-------------- + +.. note:: The `refc` GC is incremental, thread-local and not "stop-the-world". + +--mm:refc It's a deferred reference counting based garbage collector + with a simple Mark&Sweep backup GC in order to collect cycles. + Heaps are thread-local. [This document](refc.html) contains further information. +--mm:markAndSweep Simple Mark-And-Sweep based garbage collector. + Heaps are thread-local. +--mm:boehm Boehm based garbage collector, it offers a shared heap. +--mm:go Go's garbage collector, useful for interoperability with Go. + Offers a shared heap. + +--mm:none No memory management strategy nor a garbage collector. Allocated memory is + simply never freed. You should use `--mm:arc` instead. + +Here is a comparison of the different memory management modes: + +================== ======== ================= ============== ====== =================== =================== +Memory Management Heap Reference Cycles Stop-The-World Atomic Valgrind compatible Command line switch +================== ======== ================= ============== ====== =================== =================== +ORC Shared Cycle Collector No No Yes `--mm:orc` +ARC Shared Leak No No Yes `--mm:arc` +Atomic ARC Shared Leak No Yes Yes `--mm:atomicArc` +RefC Local Cycle Collector No No No `--mm:refc` +Mark & Sweep Local Cycle Collector No No No `--mm:markAndSweep` +Boehm Shared Cycle Collector Yes No No `--mm:boehm` +Go Shared Cycle Collector Yes No No `--mm:go` +None Manual Manual Manual Manual Manual `--mm:none` +================== ======== ================= ============== ====== =================== =================== + +.. default-role:: code +.. include:: rstcommon.rst + +JavaScript's garbage collector is used for the [JavaScript and NodeJS]( +backends.html#backends-the-javascript-target) compilation targets. +The [NimScript](nims.html) target uses the memory management strategy built into +the Nim compiler. diff --git a/doc/nep1.rst b/doc/nep1.md index 75343c85b..3d2a0cef3 100644 --- a/doc/nep1.rst +++ b/doc/nep1.md @@ -1,9 +1,12 @@ -============================================== -Nim Enhancement Proposal #1 - Standard Library Style Guide -============================================== +============================ +Standard Library Style Guide +============================ + :Author: Clay Sweetser, Dominik Picheta :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: @@ -18,7 +21,7 @@ library should follow. Note that there can be exceptions to these rules. Nim being as flexible as it is, there will be parts of this style guide that don't make sense in certain contexts. Furthermore, just as -`Python's style guide<http://legacy.python.org/dev/peps/pep-0008/>`_ changes +[Python's style guide](http://legacy.python.org/dev/peps/pep-0008/) changes over time, this style guide will too. These rules will only be enforced for contributions to the Nim @@ -46,7 +49,7 @@ Spacing and Whitespace Conventions Not all editors support automatic alignment of code sections, and re-aligning long sections of code by hand can quickly become tedious. - .. code-block:: nim + ```nim # This is bad, as the next time someone comes # to edit this code block, they # must re-align all the assignments again: @@ -57,25 +60,17 @@ Spacing and Whitespace Conventions CalId* = int LongLong* = int64 LongLongPtr* = ptr LongLong + ``` Naming Conventions ------------------ -Note: While the rules outlined below are the *current* naming conventions, -these conventions have not always been in place. Previously, the naming -conventions for identifiers followed the Pascal tradition of prefixes which -indicated the base type of the identifier - PFoo for pointer and reference -types, TFoo for value types, EFoo for exceptions, etc. Though this has since -changed, there are many places in the standard library which still use this -convention. Such style remains in place purely for legacy reasons, and will be -changed in the future. - - Type identifiers should be in PascalCase. All other identifiers should be in camelCase with the exception of constants which **may** use PascalCase but are not required to. - .. code-block:: nim + ```nim # Constants can start with either a lower case or upper case letter. const aConstant = 42 const FooBar = 4.2 @@ -85,6 +80,7 @@ changed in the future. # Types must start with an uppercase letter. type FooBar = object + ``` For constants coming from a C/C++ wrapper, ALL_UPPERCASE are allowed, but ugly. (Why shout CONSTANT? Constants do no harm, variables do!) @@ -95,49 +91,53 @@ changed in the future. that will be used the most, add the suffixes to the pointer variants only. The same applies to C/C++ wrappers. - .. code-block:: nim + ```nim type Handle = object # Will be used most often fd: int64 HandleRef = ref Handle # Will be used less often + ``` - Exception and Error types should have the "Error" or "Defect" suffix. - .. code-block:: nim + ```nim type ValueError = object of CatchableError AssertionDefect = object of Defect Foo = object of Exception # bad style, try to inherit CatchableError or Defect + ``` - Unless marked with the `{.pure.}` pragma, members of enums should have an identifying prefix, such as an abbreviation of the enum's name. - .. code-block:: nim + ```nim type PathComponent = enum pcDir pcLinkToDir pcFile pcLinkToFile + ``` - Non-pure enum values should use camelCase whereas pure enum values should use PascalCase. - .. code-block:: nim + ```nim type PathComponent {.pure.} = enum Dir LinkToDir File LinkToFile + ``` - In the age of HTTP, HTML, FTP, TCP, IP, UTF, WWW it is foolish to pretend these are somewhat special words requiring all uppercase. Instead treat them - as what they are: Real words. So it's ``parseUrl`` rather than - ``parseURL``, ``checkHttpHeader`` instead of ``checkHTTPHeader`` etc. + as what they are: Real words. So it's `parseUrl` rather than + `parseURL`, `checkHttpHeader` instead of `checkHTTPHeader` etc. -- Operations like ``mitems`` or ``mpairs`` (or the now deprecated ``mget``) - that allow a *mutating view* into some data structure should start with an ``m``. +- Operations like `mitems` or `mpairs` (or the now deprecated `mget`) + that allow a *mutating view* into some data structure should start with an `m`. - When both in-place mutation and 'returns transformed copy' are available the latter is a past participle of the former: @@ -145,11 +145,11 @@ changed in the future. - sort and sorted - rotate and rotated -- When the 'returns transformed copy' version already exists like ``strutils.replace`` - an in-place version should get an ``-In`` suffix (``replaceIn`` for this example). +- When the 'returns transformed copy' version already exists like `strutils.replace` + an in-place version should get an ``-In`` suffix (`replaceIn` for this example). -- Use `subjectVerb`, not `verbSubject`, eg: `fileExists`, not `existsFile`. +- Use `subjectVerb`, not `verbSubject`, e.g.: `fileExists`, not `existsFile`. The stdlib API is designed to be **easy to use** and consistent. Ease of use is measured by the number of calls to achieve a concrete high level action. The @@ -159,25 +159,29 @@ The library uses a simple naming scheme that makes use of common abbreviations to keep the names short but meaningful. -------------------- ------------ -------------------------------------- +=================== ============ ====================================== English word To use Notes -------------------- ------------ -------------------------------------- -initialize initT ``init`` is used to create a - value type ``T`` -new newP ``new`` is used to create a - reference type ``P`` +=================== ============ ====================================== +initialize initFoo initializes a value type `Foo` +new newFoo initializes a reference type `Foo` + via `new` or a value type `Foo` + with reference semantics. +this or self self for method like procs, e.g.: + `proc fun(self: Foo, a: int)` + rationale: `self` is more unique in English + than `this`, and `foo` would not be DRY. find find should return the position where something was found; for a bool result - use ``contains`` -contains contains often short for ``find() >= 0`` -append add use ``add`` instead of ``append`` + use `contains` +contains contains often short for `find() >= 0` +append add use `add` instead of `append` compare cmp should return an int with the - ``< 0`` ``== 0`` or ``> 0`` semantics; - for a bool result use ``sameXYZ`` -put put, ``[]=`` consider overloading ``[]=`` for put -get get, ``[]`` consider overloading ``[]`` for get; - consider to not use ``get`` as a - prefix: ``len`` instead of ``getLen`` + `< 0` `== 0` or `> 0` semantics; + for a bool result use `sameXYZ` +put put, `[]=` consider overloading `[]=` for put +get get, `[]` consider overloading `[]` for get; + consider to not use `get` as a + prefix: `len` instead of `getLen` length len also used for *number of elements* size size, len size should refer to a byte size capacity cap @@ -222,28 +226,29 @@ literal lit string str identifier ident indentation indent -------------------- ------------ -------------------------------------- +=================== ============ ====================================== Coding Conventions ------------------ -- The 'return' statement should ideally be used when its control-flow properties - are required. Use a procedure's implicit 'result' variable whenever possible. +- The `return` statement should ideally be used when its control-flow properties + are required. Use a procedure's implicit `result` variable whenever possible. This improves readability. - .. code-block:: nim + ```nim proc repeat(text: string, x: int): string = result = "" - for i in 0 .. x: + for i in 0..x: result.add($i) + ``` - Use a proc when possible, only using the more powerful facilities of macros, templates, iterators, and converters when necessary. -- Use the ``let`` statement (not the ``var`` statement) when declaring variables that - do not change within their scope. Using the ``let`` statement ensures that +- Use the `let` statement (not the `var` statement) when declaring variables that + do not change within their scope. Using the `let` statement ensures that variables remain immutable, and gives those who read the code a better idea of the code's purpose. @@ -251,29 +256,80 @@ Coding Conventions Conventions for multi-line statements and expressions ----------------------------------------------------- -- Tuples which are longer than one line should indent their parameters to - align with the parameters above it. +- Tuples which are longer than one line should indent their parameters. - .. code-block:: nim + ```nim type - LongTupleA = tuple[wordyTupleMemberOne: int, wordyTupleMemberTwo: string, - wordyTupleMemberThree: float] + LongTupleA = tuple[ + wordyTupleMemberOne: int, wordyTupleMemberTwo: string, + wordyTupleMemberThree: float] + ``` - Similarly, any procedure and procedure type declarations that are longer - than one line should do the same thing. + than one line should do the same thing. Double indent may be used to + distinguish them from the body that follows - this applies to all constructs + with a body (if, while, etc). - .. code-block:: nim + ```nim type - EventCallback = proc (timeReceived: Time, errorCode: int, event: Event, - output: var string) - - proc lotsOfArguments(argOne: string, argTwo: int, argThree: float, - argFour: proc(), argFive: bool): int - {.heyLookALongPragma.} = - -- Multi-line procedure calls should continue on the same column as the opening - parenthesis (like multi-line procedure declarations). - - .. code-block:: nim - startProcess(nimExecutable, currentDirectory, compilerArguments - environment, processOptions) + EventCallback = proc( + timeReceived: Time, errorCode: int, event: Event, + output: var string) + + proc lotsOfArguments( + argOne: string, argTwo: int, argThree: float, + argFour: proc(), argFive: bool, argSix: int + ): GenericType[int, string] {.heyLookALongPragma.} = + discard + ``` + +- Multi-line procedure calls should continue indented (like multi-line procedure + declarations). + + ```nim + startProcess( + nimExecutable, currentDirectory, compilerArguments + environment, processOptions) + ``` + +Previous versions of this guide advocated vertical alignment along the opening +brace / parenthesis - both styles are permissible with a preference for the +current style in new code. + +Miscellaneous +------------- + +- Use `a..b` instead of `a .. b`, except when `b` contains an operator, for example `a .. -3`. + Likewise with `a..<b`, `a..^b` and other operators starting with `..`. + +- Use `std` prefix for standard library modules, namely use `std/os` for single module and + use `std/[os, sysrand, posix]` for multiple modules. + +- Prefer multiline triple quote literals to start with a newline; it's semantically identical + (it's a feature of triple quote literals) but clearer because it aligns with the next line: + + use this: + + ```nim + let a = """ + foo + bar + """ + ``` + + instead of: + + ```nim + let a = """foo + bar + """ + ``` + +- A getter API for a private field `foo` should preferably be named `foo`, not `getFoo`. + A getter-like API should preferably be named `getFoo`, not `foo` if: + * the API has side effects + * or the cost is not `O(1)` + For in between cases, there is no clear guideline. + +- Likewise with a setter API, replacing `foo` with `foo=` and `getFoo` with `setFoo` + in the above text. diff --git a/doc/nimc.md b/doc/nimc.md new file mode 100644 index 000000000..38558454b --- /dev/null +++ b/doc/nimc.md @@ -0,0 +1,844 @@ +=================================== + Nim Compiler User Guide +=================================== + +:Author: Andreas Rumpf +:Version: |nimversion| + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +.. + +> "Look at you, hacker. A pathetic creature of meat and bone, panting and +> sweating as you run through my corridors. How can you challenge a perfect, +> immortal machine?" + + +Introduction +============ + +This document describes the usage of the *Nim compiler* +on the different supported platforms. It is not a definition of the Nim +programming language (which is covered in the [manual](manual.html)). + +Nim is free software; it is licensed under the +[MIT License](http://www.opensource.org/licenses/mit-license.php). + + +Compiler Usage +============== + +Command-line switches +--------------------- +All options that take a `PATH` or `DIR` argument are subject to path substitution: + +- `$nim`: The global nim prefix path +- `$lib`: The stdlib path +- `$home` and `~`: The user's home path +- `$config`: The directory of the module currently being compiled +- `$projectname`: The project file's name without file extension +- `$projectpath` and `$projectdir`: The project file's path +- `$nimcache`: The nimcache path + + +Basic command-line switches are: + +.. no syntax highlighting in the below included files at the moment +.. default-role:: code + +Usage: + +.. include:: basicopt.txt + +---- + +Advanced command-line switches are: + +.. include:: advopt.txt + + +.. include:: rstcommon.rst + +List of warnings +---------------- + +Each warning can be activated individually with `--warning:NAME:on|off`:option: or +in a `push` pragma with `{.warning[NAME]:on|off.}`. + +========================== ============================================ +Name Description +========================== ============================================ +CannotOpenFile Some file not essential for the compiler's + working could not be opened. +OctalEscape The code contains an unsupported octal + sequence. +Deprecated The code uses a deprecated symbol. +ConfigDeprecated The project makes use of a deprecated config + file. +SmallLshouldNotBeUsed The letter 'l' should not be used as an + identifier. +EachIdentIsTuple The code contains a confusing `var` + declaration. +CStringConv Warn about dangerous implicit conversions + to `cstring`. +EnumConv Warn about conversions from enum to enum. +AnyEnumConv Warn about any conversions to an enum type. +HoleEnumConv Warn about conversion to an enum with + holes. These conversions are unsafe. +ResultUsed Warn about the usage of the + built-in `result` variable. +User Some user-defined warning. +========================== ============================================ + + +List of hints +------------- + +Each hint can be activated individually with `--hint:NAME:on|off`:option: or in a +`push` pragma with `{.hint[NAME]:on|off.}`. + +========================== ============================================ +Name Description +========================== ============================================ +CC Shows when the C compiler is called. +CodeBegin +CodeEnd +CondTrue +Conf A config file was loaded. +ConvToBaseNotNeeded +ConvFromXtoItselfNotNeeded +Dependency +Exec Program is executed. +ExprAlwaysX +ExtendedContext +GCStats Dumps statistics about the Garbage Collector. +GlobalVar Shows global variables declarations. +Link Linking phase. +Name +Path Search paths modifications. +Pattern +Performance +Processing Artifact being compiled. +QuitCalled +Source The source line that triggered a diagnostic + message. +StackTrace +Success, SuccessX Successful compilation of a library or a binary. +User +UserRaw +XDeclaredButNotUsed Unused symbols in the code. +========================== ============================================ + + +Verbosity levels +---------------- + +===== ============================================ +Level Description +===== ============================================ +0 Minimal output level for the compiler. +1 Displays compilation of all the compiled files, including those imported + by other modules or through the [compile pragma]( + manual.html#implementation-specific-pragmas-compile-pragma). + This is the default level. +2 Displays compilation statistics, enumerates the dynamic + libraries that will be loaded by the final binary, and dumps to + standard output the result of applying [a filter to the source code]( + filters.html) if any filter was used during compilation. +3 In addition to the previous levels dumps a debug stack trace + for compiler developers. +===== ============================================ + + +Compile-time symbols +-------------------- + +Through the `-d:x`:option: or `--define:x`:option: switch you can define compile-time +symbols for conditional compilation. The defined switches can be checked in +source code with the [when statement]( +manual.html#statements-and-expressions-when-statement) and +[defined proc](system.html#defined,untyped). The typical use of this switch is +to enable builds in release mode (`-d:release`:option:) where optimizations are +enabled for better performance. Another common use is the `-d:ssl`:option: switch to +activate SSL sockets. + +Additionally, you may pass a value along with the symbol: `-d:x=y`:option: +which may be used in conjunction with the [compile-time define +pragmas](manual.html#implementation-specific-pragmas-compileminustime-define-pragmas) +to override symbols during build time. + +Compile-time symbols are completely **case insensitive** and underscores are +ignored too. `--define:FOO`:option: and `--define:foo`:option: are identical. + +Compile-time symbols starting with the `nim` prefix are reserved for the +implementation and should not be used elsewhere. + +========================== ============================================ +Name Description +========================== ============================================ +nimStdSetjmp Use the standard `setjmp()/longjmp()` library + functions for setjmp-based exceptions. This is + the default on most platforms. +nimSigSetjmp Use `sigsetjmp()/siglongjmp()` for setjmp-based exceptions. +nimRawSetjmp Use `_setjmp()/_longjmp()` on POSIX and `_setjmp()/longjmp()` + on Windows, for setjmp-based exceptions. It's the default on + BSDs and BSD-like platforms, where it's significantly faster + than the standard functions. +nimBuiltinSetjmp Use `__builtin_setjmp()/__builtin_longjmp()` for setjmp-based + exceptions. This will not work if an exception is being thrown + and caught inside the same procedure. Useful for benchmarking. +========================== ============================================ + + +Configuration files +------------------- + +**Note:** The *project file name* is the name of the ``.nim`` file that is +passed as a command-line argument to the compiler. + + +The `nim`:cmd: executable processes configuration files in the following +directories (in this order; later files overwrite previous settings): + +1) ``$nim/config/nim.cfg``, ``/etc/nim/nim.cfg`` (UNIX) or + ``<Nim's installation directory>\config\nim.cfg`` (Windows). + This file can be skipped with the `--skipCfg`:option: command line option. +2) If environment variable `XDG_CONFIG_HOME` is defined, + ``$XDG_CONFIG_HOME/nim/nim.cfg`` or ``~/.config/nim/nim.cfg`` (POSIX) or + ``%APPDATA%/nim/nim.cfg`` (Windows). + This file can be skipped with the `--skipUserCfg`:option: command line + option. +3) ``$parentDir/nim.cfg`` where ``$parentDir`` stands for any parent + directory of the project file's path. + These files can be skipped with the `--skipParentCfg`:option: + command-line option. +4) ``$projectDir/nim.cfg`` where ``$projectDir`` stands for the project + file's path. + This file can be skipped with the `--skipProjCfg`:option: + command-line option. +5) A project can also have a project-specific configuration file named + ``$project.nim.cfg`` that resides in the same directory as ``$project.nim``. + This file can be skipped with the `--skipProjCfg`:option: + command-line option. + + +Command-line settings have priority over configuration file settings. + +The default build of a project is a `debug build`:idx:. To compile a +`release build`:idx: define the `release` symbol: + + ```cmd + nim c -d:release myproject.nim + ``` + +To compile a `dangerous release build`:idx: define the `danger` symbol: + + ```cmd + nim c -d:danger myproject.nim + ``` + + +Search path handling +-------------------- + +Nim has the concept of a global search path (PATH) that is queried to +determine where to find imported modules or include files. If multiple files are +found an ambiguity error is produced. + +`nim dump`:cmd: shows the contents of the PATH. + +However before the PATH is used the current directory is checked for the +file's existence. So if PATH contains ``$lib`` and ``$lib/bar`` and the +directory structure looks like this: + + $lib/x.nim + $lib/bar/x.nim + foo/x.nim + foo/main.nim + other.nim + +And `main` imports `x`, `foo/x` is imported. If `other` imports `x` +then both ``$lib/x.nim`` and ``$lib/bar/x.nim`` match but ``$lib/x.nim`` is used +as it is the first match. + + +Generated C code directory +-------------------------- +The generated files that Nim produces all go into a subdirectory called +``nimcache``. Its full path is + +- ``$XDG_CACHE_HOME/nim/$projectname(_r|_d)`` or ``~/.cache/nim/$projectname(_r|_d)`` + on Posix +- ``$HOME\nimcache\$projectname(_r|_d)`` on Windows. + +The `_r` suffix is used for release builds, `_d` is for debug builds. + +This makes it easy to delete all generated files. + +The `--nimcache`:option: +[compiler switch][command-line switches] can be used to +to change the ``nimcache`` directory. + +However, the generated C code is not platform-independent. C code generated for +Linux does not compile on Windows, for instance. The comment on top of the +C file lists the OS, CPU, and CC the file has been compiled for. + + +Compiler Selection +================== + +To change the compiler from the default compiler (at the command line): + + ```cmd + nim c --cc:llvm_gcc --compile_only myfile.nim + ``` + +This uses the configuration defined in ``config\nim.cfg`` for `llvm_gcc`:cmd:. + +If nimcache already contains compiled code from a different compiler for the same project, +add the `-f`:option: flag to force all files to be recompiled. + +The default compiler is defined at the top of ``config\nim.cfg``. +Changing this setting affects the compiler used by `koch`:cmd: to (re)build Nim. + +To use the `CC` environment variable, use `nim c --cc:env myfile.nim`:cmd:. +To use the `CXX` environment variable, use `nim cpp --cc:env myfile.nim`:cmd:. +`--cc:env`:option: is available since Nim version 1.4. + + +Cross-compilation +================= + +To cross compile, use for example: + + ```cmd + nim c --cpu:i386 --os:linux --compileOnly --genScript myproject.nim + ``` + +Then move the C code and the compile script `compile_myproject.sh`:cmd: to your +Linux i386 machine and run the script. + +Another way is to make Nim invoke a cross compiler toolchain: + + ```cmd + nim c --cpu:arm --os:linux myproject.nim + ``` + +For cross compilation, the compiler invokes a C compiler named like +`$cpu.$os.$cc` (for example `arm.linux.gcc`) with options defined in +`$cpu.$os.$cc.options.always`. The configuration system is used to provide +meaningful defaults. For example, for Linux on a 32-bit ARM CPU, your +configuration file should contain something like: + + arm.linux.gcc.path = "/usr/bin" + arm.linux.gcc.exe = "arm-linux-gcc" + arm.linux.gcc.linkerexe = "arm-linux-gcc" + arm.linux.gcc.options.always = "-w -fmax-errors=3" + +Cross-compilation for Windows +============================= + +To cross-compile for Windows from Linux or macOS using the MinGW-w64 toolchain: + + ```cmd + nim c -d:mingw myproject.nim + # `nim r` also works, running the binary via `wine` or `wine64`: + nim r -d:mingw --eval:'import os; echo "a" / "b"' + ``` + +Use `--cpu:i386`:option: or `--cpu:amd64`:option: to switch the CPU architecture. + +The MinGW-w64 toolchain can be installed as follows: + + ```cmd + apt install mingw-w64 # Ubuntu + yum install mingw32-gcc + yum install mingw64-gcc # CentOS - requires EPEL + brew install mingw-w64 # OSX + ``` + + +Cross-compilation for Android +============================= + +There are two ways to compile for Android: terminal programs (Termux) and with +the NDK (Android Native Development Kit). + +The first one is to treat Android as a simple Linux and use +[Termux](https://wiki.termux.com) to connect and run the Nim compiler +directly on android as if it was Linux. These programs are console-only +programs that can't be distributed in the Play Store. + +Use regular `nim c`:cmd: inside termux to make Android terminal programs. + +Normal Android apps are written in Java, to use Nim inside an Android app +you need a small Java stub that calls out to a native library written in +Nim using the [NDK](https://developer.android.com/ndk). You can also use +[native-activity](https://developer.android.com/ndk/samples/sample_na) +to have the Java stub be auto-generated for you. + +Use `nim c -c --cpu:arm --os:android -d:androidNDK --noMain:on`:cmd: to +generate the C source files you need to include in your Android Studio +project. Add the generated C files to CMake build script in your Android +project. Then do the final compile with Android Studio which uses Gradle +to call CMake to compile the project. + +Because Nim is part of a library it can't have its own C-style `main()`:c: +so you would need to define your own `android_main`:c: and init the Java +environment, or use a library like SDL2 or GLFM to do it. After the Android +stuff is done, it's very important to call `NimMain()`:c: in order to +initialize Nim's garbage collector and to run the top level statements +of your program. + + ```Nim + proc NimMain() {.importc.} + proc glfmMain*(display: ptr GLFMDisplay) {.exportc.} = + NimMain() # initialize garbage collector memory, types and stack + ``` + + +The name `NimMain` can be influenced via the `--nimMainPrefix:prefix` switch. +Use `--nimMainPrefix:MyLib` and the function to call is named `MyLibNimMain`. + + +Cross-compilation for iOS +========================= + +To cross-compile for iOS you need to be on a macOS computer and use XCode. +Normal languages for iOS development are Swift and Objective C. Both of these +use LLVM and can be compiled into object files linked together with C, C++ +or Objective C code produced by Nim. + +Use `nim c -c --os:ios --noMain:on`:cmd: to generate C files and include them in +your XCode project. Then you can use XCode to compile, link, package and +sign everything. + +Because Nim is part of a library it can't have its own C-style `main()`:c: so you +would need to define `main` that calls `autoreleasepool` and +`UIApplicationMain` to do it, or use a library like SDL2 or GLFM. After +the iOS setup is done, it's very important to call `NimMain()`:c: to +initialize Nim's garbage collector and to run the top-level statements +of your program. + + ```Nim + proc NimMain() {.importc.} + proc glfmMain*(display: ptr GLFMDisplay) {.exportc.} = + NimMain() # initialize garbage collector memory, types and stack + ``` + +Note: XCode's "make clean" gets confused about the generated nim.c files, +so you need to clean those files manually to do a clean build. + +The name `NimMain` can be influenced via the `--nimMainPrefix:prefix` switch. +Use `--nimMainPrefix:MyLib` and the function to call is named `MyLibNimMain`. + + +Cross-compilation for Nintendo Switch +===================================== + +Simply add `--os:nintendoswitch`:option: +to your usual `nim c`:cmd: or `nim cpp`:cmd: command and set the `passC`:option: +and `passL`:option: command line switches to something like: + + ```cmd + nim c ... --d:nimAllocPagesViaMalloc --mm:orc --passC="-I$DEVKITPRO/libnx/include" ... + --passL="-specs=$DEVKITPRO/libnx/switch.specs -L$DEVKITPRO/libnx/lib -lnx" + ``` + +or setup a ``nim.cfg`` file like so: + + #nim.cfg + --mm:orc + --d:nimAllocPagesViaMalloc + --define:nimInheritHandles + --passC="-I$DEVKITPRO/libnx/include" + --passL="-specs=$DEVKITPRO/libnx/switch.specs -L$DEVKITPRO/libnx/lib -lnx" + +The devkitPro setup must be the same as the default with their new installer +[here for Mac/Linux](https://github.com/devkitPro/pacman/releases) or +[here for Windows](https://github.com/devkitPro/installer/releases). + +For example, with the above-mentioned config: + + ```cmd + nim c --os:nintendoswitch switchhomebrew.nim + ``` + +This will generate a file called ``switchhomebrew.elf`` which can then be turned into +an nro file with the `elf2nro`:cmd: tool in the devkitPro release. Examples can be found at +[the nim-libnx github repo](https://github.com/jyapayne/nim-libnx.git). + +There are a few things that don't work because the devkitPro libraries don't support them. +They are: + +1. Waiting for a subprocess to finish. A subprocess can be started, but right + now it can't be waited on, which sort of makes subprocesses a bit hard to use +2. Dynamic calls. Switch OS (Horizon) doesn't support dynamic libraries, so dlopen/dlclose are not available. +3. mqueue. Sadly there are no mqueue headers. +4. ucontext. No headers for these either. No coroutines for now :( +5. nl_types. No headers for this. +6. As mmap is not supported, the nimAllocPagesViaMalloc option has to be used. + +GPU Compilation +=============== + +Compiling for GPU computation can be achieved with `--cc:nvcc` for CUDA with nvcc, or with `--cc:hipcc` for AMD GPUs with HIP. Both compilers require building for C++ with `nim cpp`. + +Here's a very simple CUDA kernel example using emit, which can be compiled with `nim cpp --cc:nvcc --define:"useMalloc" hello_kernel.nim` assuming you have the CUDA toolkit installed. + +```nim +{.emit: """ +__global__ void add(int a, int b) { + int c; + c = a + b; +} +""".} + +proc main() = + {.emit: """ + add<<<1,1>>>(2,7); + """.} + +main() +``` + +DLL generation +============== + +**Note**: The same rules apply to `lib*.so` shared object files on UNIX. For better +readability only the DLL version is described here. + +Nim supports the generation of DLLs. However, there must be only one +instance of the GC per process/address space. This instance is contained in +``nimrtl.dll``. This means that every generated Nim DLL depends +on ``nimrtl.dll``. To generate the "nimrtl.dll" file, use the command: + + ```cmd + nim c -d:release lib/nimrtl.nim + ``` + +To link against ``nimrtl.dll`` use the command: + + ```cmd + nim c -d:useNimRtl myprog.nim + ``` + + +Additional compilation switches +=============================== + +The standard library supports a growing number of `useX` conditional defines +affecting how some features are implemented. This section tries to give a +complete list. + +====================== ========================================================= +Define Effect +====================== ========================================================= +`release` Turns on the optimizer. + More aggressive optimizations are possible, e.g.: + `--passC:-ffast-math`:option: (but see issue #10305) +`danger` Turns off all runtime checks and turns on the optimizer. +`useFork` Makes `osproc` use `fork`:c: instead of `posix_spawn`:c:. +`useNimRtl` Compile and link against ``nimrtl.dll``. +`useMalloc` Makes Nim use C's `malloc`:idx: instead of Nim's + own memory manager, albeit prefixing each allocation with + its size to support clearing memory on reallocation. + This only works with `--mm:none`:option:, + `--mm:arc`:option: and `--mm:orc`:option:. +`useRealtimeGC` Enables support of Nim's GC for *soft* realtime + systems. See the documentation of the [refc](refc.html) + for further information. +`logGC` Enable GC logging to stdout. +`nodejs` The JS target is actually ``node.js``. +`ssl` Enables OpenSSL support for the sockets module. +`memProfiler` Enables memory profiling for the native GC. +`uClibc` Use uClibc instead of libc. (Relevant for Unix-like OSes) +`checkAbi` When using types from C headers, add checks that compare + what's in the Nim file with what's in the C header. + This may become enabled by default in the future. +`tempDir` This symbol takes a string as its value, like + `--define:tempDir:/some/temp/path`:option: to override + the temporary directory returned by `os.getTempDir()`. + The value **should** end with a directory separator + character. (Relevant for the Android platform) +`useShPath` This symbol takes a string as its value, like + `--define:useShPath:/opt/sh/bin/sh`:option: to override + the path for the `sh`:cmd: binary, in cases where it is + not located in the default location ``/bin/sh``. +`noSignalHandler` Disable the crash handler from ``system.nim``. +`globalSymbols` Load all `{.dynlib.}` libraries with the `RTLD_GLOBAL`:c: + flag on Posix systems to resolve symbols in subsequently + loaded libraries. +`lto` Enable link-time optimization in the backend compiler and + linker. +`lto_incremental` Enable link-time optimization and additionally enable + incremental linking for compilers that support it. + Currently only clang and vcc. +`strip` Strip debug symbols added by the backend compiler from + the executable. +====================== ========================================================= + + + +Additional Features +=================== + +This section describes Nim's additional features that are not listed in the +Nim manual. Some of the features here only make sense for the C code +generator and are subject to change. + + +LineDir option +-------------- +The `--lineDir`:option: option can be turned on or off. If turned on the +generated C code contains `#line`:c: directives. This may be helpful for +debugging with GDB. + + +StackTrace option +----------------- +If the `--stackTrace`:option: option is turned on, the generated C contains code to +ensure that proper stack traces are given if the program crashes or some uncaught exception is raised. + + +LineTrace option +---------------- +The `--lineTrace`:option: option implies the `stackTrace`:option: option. +If turned on, +the generated C contains code to ensure that proper stack traces with line +number information are given if the program crashes or an uncaught exception +is raised. + + +DynlibOverride +============== + +By default Nim's `dynlib` pragma causes the compiler to generate +`GetProcAddress`:cpp: (or their Unix counterparts) +calls to bind to a DLL. With the `dynlibOverride`:option: command line switch this +can be prevented and then via `--passL`:option: the static library can be linked +against. For instance, to link statically against Lua this command might work +on Linux: + + ```cmd + nim c --dynlibOverride:lua --passL:liblua.lib program.nim + ``` + + +Backend language options +======================== + +The typical compiler usage involves using the `compile`:option: or `c`:option: +command to transform a ``.nim`` file into one or more ``.c`` files which are then +compiled with the platform's C compiler into a static binary. However, there +are other commands to compile to C++, Objective-C, or JavaScript. More details +can be read in the [Nim Backend Integration document](backends.html). + + +Nim documentation tools +======================= + +Nim provides the `doc`:idx: command to generate HTML +documentation from ``.nim`` source files. Only exported symbols will appear in +the output. For more details see [the docgen documentation](docgen.html). + +Nim idetools integration +======================== + +Nim provides language integration with external IDEs through the +idetools command. See the documentation of [idetools](idetools.html) +for further information. + +.. + Nim interactive mode + ==================== + + The Nim compiler supports an interactive mode. This is also known as + a `REPL`:idx: (*read eval print loop*). If Nim has been built with the + `-d:nimUseLinenoise` switch, it uses the GNU readline library for terminal + input management. To start Nim in interactive mode use the command + `nim secret`. To quit use the `quit()` command. To determine whether an input + line is an incomplete statement to be continued these rules are used: + + 1. The line ends with ``[-+*/\\<>!\?\|%&$@~,;:=#^]\s*$`` (operator symbol followed by optional whitespace). + 2. The line starts with a space (indentation). + 3. The line is within a triple quoted string literal. However, the detection + does not work if the line contains more than one `"""`. + + +Nim for embedded systems +======================== + +While the default Nim configuration is targeted for optimal performance on +modern PC hardware and operating systems with ample memory, it is very well +possible to run Nim code and a good part of the Nim standard libraries on small +embedded microprocessors with only a few kilobytes of memory. + +A good start is to use the `any` operating target together with the +`malloc` memory allocator and the `arc` garbage collector. For example: + + ```cmd + nim c --os:any --mm:arc -d:useMalloc [...] x.nim + ``` + +- `--mm:arc`:option: will enable the reference counting memory management instead + of the default garbage collector. This enables Nim to use heap memory which + is required for strings and seqs, for example. + +- The `--os:any`:option: target makes sure Nim does not depend on any specific + operating system primitives. Your platform should support only some basic + ANSI C library `stdlib` and `stdio` functions which should be available + on almost any platform. + +- The `-d:useMalloc`:option: option configures Nim to use only the standard C memory + manage primitives `malloc()`:c:, `free()`:c:, `realloc()`:c:. + +If your platform does not provide these functions it should be trivial to +provide an implementation for them and link these to your program. + +For targets with very restricted memory, it might be beneficial to pass some +additional flags to both the Nim compiler and the C compiler and/or linker +to optimize the build for size. For example, the following flags can be used +when targeting a gcc compiler: + +`--opt:size -d:lto -d:strip`:option: + +The `--opt:size`:option: flag instructs Nim to optimize code generation for small +size (with the help of the C compiler), the `-d:lto`:option: flags enable link-time +optimization in the compiler and linker, the `-d:strip`:option: strips debug symbols. + +Check the [Cross-compilation] section for instructions on how to compile the +program for your target. + + +nimAllocPagesViaMalloc +---------------------- + +Nim's default allocator is based on TLSF, this algorithm was designed for embedded +devices. This allocator gets blocks/pages of memory via a currently undocumented +`osalloc` API which usually uses POSIX's `mmap` call. On many environments `mmap` +is not available but C's `malloc` is. You can use the `nimAllocPagesViaMalloc` +define to use `malloc` instead of `mmap`. `nimAllocPagesViaMalloc` is currently +only supported with `--mm:arc` or `--mm:orc`. (Since version 1.6) + +nimPage256 / nimPage512 / nimPage1k +=================================== + +Adjust the page size for Nim's GC allocator. This enables using +`nimAllocPagesViaMalloc` on devices with less RAM. The default +page size requires too much RAM to work. + +Recommended settings: + +- < 32 kB of RAM use `nimPage256` + +- < 512 kB of RAM use `nimPage512` + +- < 2 MB of RAM use `nimPage1k` + +Initial testing hasn't shown much difference between 512B or 1kB page sizes +in terms of performance or latency. Using `nimPages256` will limit the +total amount of allocatable RAM. + +nimMemAlignTiny +=============== + +Sets `MemAlign` to `4` bytes which reduces the memory alignment +to better match some embedded devices. + +Thread stack size +================= + +Nim's thread API provides a simple wrapper around more advanced +RTOS task features. Customizing the stack size and stack guard size can +be done by setting `-d:nimThreadStackSize=16384` or `-d:nimThreadStackGuard=32`. + +Currently only Zephyr, NuttX and FreeRTOS support these configurations. + +Nim for realtime systems +======================== + +See the `--mm:arc` or `--mm:orc` memory management settings in +[MM](mm.html) for further information. + + +Signal handling in Nim +====================== + +The Nim programming language has no concept of Posix's signal handling +mechanisms. However, the standard library offers some rudimentary support +for signal handling, in particular, segmentation faults are turned into +fatal errors that produce a stack trace. This can be disabled with the +`-d:noSignalHandler`:option: switch. + + +Optimizing for Nim +================== + +Nim has no separate optimizer, but the C code that is produced is very +efficient. Most C compilers have excellent optimizers, so usually it is +not needed to optimize one's code. Nim has been designed to encourage +efficient code: The most readable code in Nim is often the most efficient +too. + +However, sometimes one has to optimize. Do it in the following order: + +1. switch off the embedded debugger (it is **slow**!) +2. turn on the optimizer and turn off runtime checks +3. profile your code to find where the bottlenecks are +4. try to find a better algorithm +5. do low-level optimizations + +This section can only help you with the last item. + + +Optimizing string handling +-------------------------- + +String assignments are sometimes expensive in Nim: They are required to +copy the whole string. However, the compiler is often smart enough to not copy +strings. Due to the argument passing semantics, strings are never copied when +passed to subroutines. The compiler does not copy strings that are a result of +a procedure call, because the callee returns a new string anyway. +Thus it is efficient to do: + + ```Nim + var s = procA() # assignment will not copy the string; procA allocates a new + # string already + ``` + +However, it is not efficient to do: + + ```Nim + var s = varA # assignment has to copy the whole string into a new buffer! + ``` + +For `let` symbols a copy is not always necessary: + + ```Nim + let s = varA # may only copy a pointer if it safe to do so + ``` + +The compiler optimizes string case statements: A hashing scheme is used for them +if several different string constants are used. So code like this is reasonably +efficient: + + ```Nim + case normalize(k.key) + of "name": c.name = v + of "displayname": c.displayName = v + of "version": c.version = v + of "os": c.oses = split(v, {';'}) + of "cpu": c.cpus = split(v, {';'}) + of "authors": c.authors = split(v, {';'}) + of "description": c.description = v + of "app": + case normalize(v) + of "console": c.app = appConsole + of "gui": c.app = appGUI + else: quit(errorStr(p, "expected: console or gui")) + of "license": c.license = UnixToNativePath(k.value) + else: quit(errorStr(p, "unknown variable: " & k.key)) + ``` diff --git a/doc/nimc.rst b/doc/nimc.rst deleted file mode 100644 index bd5e897c3..000000000 --- a/doc/nimc.rst +++ /dev/null @@ -1,680 +0,0 @@ -=================================== - Nim Compiler User Guide -=================================== - -:Author: Andreas Rumpf -:Version: |nimversion| - -.. contents:: - - "Look at you, hacker. A pathetic creature of meat and bone, panting and - sweating as you run through my corridors. How can you challenge a perfect, - immortal machine?" - - -Introduction -============ - -This document describes the usage of the *Nim compiler* -on the different supported platforms. It is not a definition of the Nim -programming language (which is covered in the `manual <manual.html>`_). - -Nim is free software; it is licensed under the -`MIT License <http://www.opensource.org/licenses/mit-license.php>`_. - - -Compiler Usage -============== - -Command line switches ---------------------- -Basic command line switches are: - -Usage: - -.. include:: basicopt.txt - ----- - -Advanced command line switches are: - -.. include:: advopt.txt - - - -List of warnings ----------------- - -Each warning can be activated individually with ``--warning[NAME]:on|off`` or -in a ``push`` pragma. - -========================== ============================================ -Name Description -========================== ============================================ -CannotOpenFile Some file not essential for the compiler's - working could not be opened. -OctalEscape The code contains an unsupported octal - sequence. -Deprecated The code uses a deprecated symbol. -ConfigDeprecated The project makes use of a deprecated config - file. -SmallLshouldNotBeUsed The letter 'l' should not be used as an - identifier. -EachIdentIsTuple The code contains a confusing ``var`` - declaration. -User Some user defined warning. -========================== ============================================ - - -List of hints -------------- - -Each hint can be activated individually with ``--hint[NAME]:on|off`` or in a -``push`` pragma. - -========================== ============================================ -Name Description -========================== ============================================ -CC Shows when the C compiler is called. -CodeBegin -CodeEnd -CondTrue -Conf A config file was loaded. -ConvToBaseNotNeeded -ConvFromXtoItselfNotNeeded -Dependency -Exec Program is executed. -ExprAlwaysX -ExtendedContext -GCStats Dumps statistics about the Garbage Collector. -GlobalVar Shows global variables declarations. -LineTooLong Line exceeds the maximum length. -Link Linking phase. -Name -Path Search paths modifications. -Pattern -Performance -Processing Artifact being compiled. -QuitCalled -Source The source line that triggered a diagnostic - message. -StackTrace -Success, SuccessX Successful compilation of a library or a binary. -User -UserRaw -XDeclaredButNotUsed Unused symbols in the code. -========================== ============================================ - - -Verbosity levels ----------------- - -===== ============================================ -Level Description -===== ============================================ -0 Minimal output level for the compiler. -1 Displays compilation of all the compiled files, including those imported - by other modules or through the `compile pragma - <manual.html#implementation-specific-pragmas-compile-pragma>`_. - This is the default level. -2 Displays compilation statistics, enumerates the dynamic - libraries that will be loaded by the final binary and dumps to - standard output the result of applying `a filter to the source code - <filters.html>`_ if any filter was used during compilation. -3 In addition to the previous levels dumps a debug stack trace - for compiler developers. -===== ============================================ - - -Compile time symbols --------------------- - -Through the ``-d:x`` or ``--define:x`` switch you can define compile time -symbols for conditional compilation. The defined switches can be checked in -source code with the `when statement -<manual.html#statements-and-expressions-when-statement>`_ and -`defined proc <system.html#defined,untyped>`_. The typical use of this switch is -to enable builds in release mode (``-d:release``) where optimizations are -enabled for better performance. Another common use is the ``-d:ssl`` switch to -activate SSL sockets. - -Additionally, you may pass a value along with the symbol: ``-d:x=y`` -which may be used in conjunction with the `compile time define -pragmas<manual.html#implementation-specific-pragmas-compile-time-define-pragmas>`_ -to override symbols during build time. - -Compile time symbols are completely **case insensitive** and underscores are -ignored too. ``--define:FOO`` and ``--define:foo`` are identical. - -Compile time symbols starting with the ``nim`` prefix are reserved for the -implementation and should not be used elsewhere. - - -Configuration files -------------------- - -**Note:** The *project file name* is the name of the ``.nim`` file that is -passed as a command line argument to the compiler. - - -The ``nim`` executable processes configuration files in the following -directories (in this order; later files overwrite previous settings): - -1) ``$nim/config/nim.cfg``, ``/etc/nim/nim.cfg`` (UNIX) or ``<Nim's installation directory>\config\nim.cfg`` (Windows). This file can be skipped with the ``--skipCfg`` command line option. -2) If environment variable ``XDG_CONFIG_HOME`` is defined, ``$XDG_CONFIG_HOME/nim/nim.cfg`` or ``~/.config/nim/nim.cfg`` (POSIX) or ``%APPDATA%/nim/nim.cfg`` (Windows). This file can be skipped with the ``--skipUserCfg`` command line option. -3) ``$parentDir/nim.cfg`` where ``$parentDir`` stands for any parent directory of the project file's path. These files can be skipped with the ``--skipParentCfg`` command line option. -4) ``$projectDir/nim.cfg`` where ``$projectDir`` stands for the project file's path. This file can be skipped with the ``--skipProjCfg`` command line option. -5) A project can also have a project specific configuration file named ``$project.nim.cfg`` that resides in the same directory as ``$project.nim``. This file can be skipped with the ``--skipProjCfg`` command line option. - - -Command line settings have priority over configuration file settings. - -The default build of a project is a `debug build`:idx:. To compile a -`release build`:idx: define the ``release`` symbol:: - - nim c -d:release myproject.nim - - To compile a `dangerous release build`:idx: define the ``danger`` symbol:: - - nim c -d:danger myproject.nim - - -Search path handling --------------------- - -Nim has the concept of a global search path (PATH) that is queried to -determine where to find imported modules or include files. If multiple files are -found an ambiguity error is produced. - -``nim dump`` shows the contents of the PATH. - -However before the PATH is used the current directory is checked for the -file's existence. So if PATH contains ``$lib`` and ``$lib/bar`` and the -directory structure looks like this:: - - $lib/x.nim - $lib/bar/x.nim - foo/x.nim - foo/main.nim - other.nim - -And ``main`` imports ``x``, ``foo/x`` is imported. If ``other`` imports ``x`` -then both ``$lib/x.nim`` and ``$lib/bar/x.nim`` match but ``$lib/x.nim`` is used -as it is the first match. - - -Generated C code directory --------------------------- -The generated files that Nim produces all go into a subdirectory called -``nimcache``. Its full path is - -- ``$XDG_CACHE_HOME/nim/$projectname(_r|_d)`` or ``~/.cache/nim/$projectname(_r|_d)`` - on Posix -- ``$HOME/nimcache/$projectname(_r|_d)`` on Windows. - -The ``_r`` suffix is used for release builds, ``_d`` is for debug builds. - -This makes it easy to delete all generated files. - -The ``--nimcache`` -`compiler switch <#compiler-usage-command-line-switches>`_ can be used to -to change the ``nimcache`` directory. - -However, the generated C code is not platform independent. C code generated for -Linux does not compile on Windows, for instance. The comment on top of the -C file lists the OS, CPU and CC the file has been compiled for. - - -Compiler Selection -================== - -To change the compiler from the default compiler (at the command line):: - - nim c --cc:llvm_gcc --compile_only myfile.nim - -This uses the configuration defined in ``config\nim.cfg`` for ``lvm_gcc``. - -If nimcache already contains compiled code from a different compiler for the same project, -add the ``-f`` flag to force all files to be recompiled. - -The default compiler is defined at the top of ``config\nim.cfg``. -Changing this setting affects the compiler used by ``koch`` to (re)build Nim. - -To use the ``CC`` environment variable, use ``nim c --cc:env myfile.nim``. To use the -``CXX`` environment variable, use ``nim cpp --cc:env myfile.nim``. ``--cc:env`` is available -since Nim version 1.4. - - -Cross compilation -================= - -To cross compile, use for example:: - - nim c --cpu:i386 --os:linux --compileOnly --genScript myproject.nim - -Then move the C code and the compile script ``compile_myproject.sh`` to your -Linux i386 machine and run the script. - -Another way is to make Nim invoke a cross compiler toolchain:: - - nim c --cpu:arm --os:linux myproject.nim - -For cross compilation, the compiler invokes a C compiler named -like ``$cpu.$os.$cc`` (for example arm.linux.gcc) and the configuration -system is used to provide meaningful defaults. For example for ``ARM`` your -configuration file should contain something like:: - - arm.linux.gcc.path = "/usr/bin" - arm.linux.gcc.exe = "arm-linux-gcc" - arm.linux.gcc.linkerexe = "arm-linux-gcc" - -Cross compilation for Windows -============================= - -To cross compile for Windows from Linux or macOS using the MinGW-w64 toolchain:: - - nim c -d:mingw myproject.nim - -Use ``--cpu:i386`` or ``--cpu:amd64`` to switch the CPU architecture. - -The MinGW-w64 toolchain can be installed as follows:: - - Ubuntu: apt install mingw-w64 - CentOS: yum install mingw32-gcc | mingw64-gcc - requires EPEL - OSX: brew install mingw-w64 - - -Cross compilation for Android -============================= - -There are two ways to compile for Android: terminal programs (Termux) and with -the NDK (Android Native Development Kit). - -First one is to treat Android as a simple Linux and use -`Termux <https://wiki.termux.com>`_ to connect and run the Nim compiler -directly on android as if it was Linux. These programs are console only -programs that can't be distributed in the Play Store. - -Use regular ``nim c`` inside termux to make Android terminal programs. - -Normal Android apps are written in Java, to use Nim inside an Android app -you need a small Java stub that calls out to a native library written in -Nim using the `NDK <https://developer.android.com/ndk>`_. You can also use -`native-acitivty <https://developer.android.com/ndk/samples/sample_na>`_ -to have the Java stub be auto generated for you. - -Use ``nim c -c --cpu:arm --os:android -d:androidNDK --noMain:on`` to -generate the C source files you need to include in your Android Studio -project. Add the generated C files to CMake build script in your Android -project. Then do the final compile with Android Studio which uses Gradle -to call CMake to compile the project. - -Because Nim is part of a library it can't have its own c style ``main()`` -so you would need to define your own ``android_main`` and init the Java -environment, or use a library like SDL2 or GLFM to do it. After the Android -stuff is done, it's very important to call ``NimMain()`` in order to -initialize Nim's garbage collector and to run the top level statements -of your program. - -.. code-block:: Nim - - proc NimMain() {.importc.} - proc glfmMain*(display: ptr GLFMDisplay) {.exportc.} = - NimMain() # initialize garbage collector memory, types and stack - - -Cross compilation for iOS -========================= - -To cross compile for iOS you need to be on a MacOS computer and use XCode. -Normal languages for iOS development are Swift and Objective C. Both of these -use LLVM and can be compiled into object files linked together with C, C++ -or Objective C code produced by Nim. - -Use ``nim c -c --os:ios --noMain:on`` to generate C files and include them in -your XCode project. Then you can use XCode to compile, link, package and -sign everything. - -Because Nim is part of a library it can't have its own c style ``main()`` so you -would need to define `main` that calls ``autoreleasepool`` and -``UIApplicationMain`` to do it, or use a library like SDL2 or GLFM. After -the iOS setup is done, it's very important to call ``NimMain()`` in order to -initialize Nim's garbage collector and to run the top level statements -of your program. - -.. code-block:: Nim - - proc NimMain() {.importc.} - proc glfmMain*(display: ptr GLFMDisplay) {.exportc.} = - NimMain() # initialize garbage collector memory, types and stack - -Note: XCode's "make clean" gets confused about the generated nim.c files, -so you need to clean those files manually to do a clean build. - - -Cross compilation for Nintendo Switch -===================================== - -Simply add --os:nintendoswitch -to your usual ``nim c`` or ``nim cpp`` command and set the ``passC`` -and ``passL`` command line switches to something like: - -.. code-block:: console - nim c ... --passC="-I$DEVKITPRO/libnx/include" ... - --passL="-specs=$DEVKITPRO/libnx/switch.specs -L$DEVKITPRO/libnx/lib -lnx" - -or setup a nim.cfg file like so: - -.. code-block:: Nim - #nim.cfg - --passC="-I$DEVKITPRO/libnx/include" - --passL="-specs=$DEVKITPRO/libnx/switch.specs -L$DEVKITPRO/libnx/lib -lnx" - -The DevkitPro setup must be the same as the default with their new installer -`here for Mac/Linux <https://github.com/devkitPro/pacman/releases>`_ or -`here for Windows <https://github.com/devkitPro/installer/releases>`_. - -For example, with the above mentioned config:: - - nim c --os:nintendoswitch switchhomebrew.nim - -This will generate a file called ``switchhomebrew.elf`` which can then be turned into -an nro file with the ``elf2nro`` tool in the DevkitPro release. Examples can be found at -`the nim-libnx github repo <https://github.com/jyapayne/nim-libnx.git>`_. - -There are a few things that don't work because the DevkitPro libraries don't support them. -They are: - -1. Waiting for a subprocess to finish. A subprocess can be started, but right - now it can't be waited on, which sort of makes subprocesses a bit hard to use -2. Dynamic calls. DevkitPro libraries have no dlopen/dlclose functions. -3. Command line parameters. It doesn't make sense to have these for a console - anyways, so no big deal here. -4. mqueue. Sadly there are no mqueue headers. -5. ucontext. No headers for these either. No coroutines for now :( -6. nl_types. No headers for this. - -DLL generation -============== - -Nim supports the generation of DLLs. However, there must be only one -instance of the GC per process/address space. This instance is contained in -``nimrtl.dll``. This means that every generated Nim DLL depends -on ``nimrtl.dll``. To generate the "nimrtl.dll" file, use the command:: - - nim c -d:release lib/nimrtl.nim - -To link against ``nimrtl.dll`` use the command:: - - nim c -d:useNimRtl myprog.nim - -**Note**: Currently the creation of ``nimrtl.dll`` with thread support has -never been tested and is unlikely to work! - - -Additional compilation switches -=============================== - -The standard library supports a growing number of ``useX`` conditional defines -affecting how some features are implemented. This section tries to give a -complete list. - -====================== ========================================================= -Define Effect -====================== ========================================================= -``release`` Turns on the optimizer. - More aggressive optimizations are possible, eg: - ``--passC:-ffast-math`` (but see issue #10305) -``danger`` Turns off all runtime checks and turns on the optimizer. -``useFork`` Makes ``osproc`` use ``fork`` instead of ``posix_spawn``. -``useNimRtl`` Compile and link against ``nimrtl.dll``. -``useMalloc`` Makes Nim use C's `malloc`:idx: instead of Nim's - own memory manager, albeit prefixing each allocation with - its size to support clearing memory on reallocation. - This only works with ``gc:none`` and - with ``--newruntime``. -``useRealtimeGC`` Enables support of Nim's GC for *soft* realtime - systems. See the documentation of the `gc <gc.html>`_ - for further information. -``logGC`` Enable GC logging to stdout. -``nodejs`` The JS target is actually ``node.js``. -``ssl`` Enables OpenSSL support for the sockets module. -``memProfiler`` Enables memory profiling for the native GC. -``uClibc`` Use uClibc instead of libc. (Relevant for Unix-like OSes) -``checkAbi`` When using types from C headers, add checks that compare - what's in the Nim file with what's in the C header. - This may become enabled by default in the future. -``tempDir`` This symbol takes a string as its value, like - ``--define:tempDir:/some/temp/path`` to override the - temporary directory returned by ``os.getTempDir()``. - The value **should** end with a directory separator - character. (Relevant for the Android platform) -``useShPath`` This symbol takes a string as its value, like - ``--define:useShPath:/opt/sh/bin/sh`` to override the - path for the ``sh`` binary, in cases where it is not - located in the default location ``/bin/sh``. -``noSignalHandler`` Disable the crash handler from ``system.nim``. -====================== ========================================================= - - - -Additional Features -=================== - -This section describes Nim's additional features that are not listed in the -Nim manual. Some of the features here only make sense for the C code -generator and are subject to change. - - -LineDir option --------------- -The ``lineDir`` option can be turned on or off. If turned on the -generated C code contains ``#line`` directives. This may be helpful for -debugging with GDB. - - -StackTrace option ------------------ -If the ``stackTrace`` option is turned on, the generated C contains code to -ensure that proper stack traces are given if the program crashes or an -uncaught exception is raised. - - -LineTrace option ----------------- -The ``lineTrace`` option implies the ``stackTrace`` option. If turned on, -the generated C contains code to ensure that proper stack traces with line -number information are given if the program crashes or an uncaught exception -is raised. - - -DynlibOverride -============== - -By default Nim's ``dynlib`` pragma causes the compiler to generate -``GetProcAddress`` (or their Unix counterparts) -calls to bind to a DLL. With the ``dynlibOverride`` command line switch this -can be prevented and then via ``--passL`` the static library can be linked -against. For instance, to link statically against Lua this command might work -on Linux:: - - nim c --dynlibOverride:lua --passL:liblua.lib program.nim - - -Backend language options -======================== - -The typical compiler usage involves using the ``compile`` or ``c`` command to -transform a ``.nim`` file into one or more ``.c`` files which are then -compiled with the platform's C compiler into a static binary. However there -are other commands to compile to C++, Objective-C or JavaScript. More details -can be read in the `Nim Backend Integration document <backends.html>`_. - - -Nim documentation tools -======================= - -Nim provides the `doc`:idx: and `doc2`:idx: commands to generate HTML -documentation from ``.nim`` source files. Only exported symbols will appear in -the output. For more details `see the docgen documentation <docgen.html>`_. - -Nim idetools integration -======================== - -Nim provides language integration with external IDEs through the -idetools command. See the documentation of `idetools <idetools.html>`_ -for further information. - -.. - Nim interactive mode - ==================== - - The Nim compiler supports an interactive mode. This is also known as - a `REPL`:idx: (*read eval print loop*). If Nim has been built with the - ``-d:nimUseLinenoise`` switch, it uses the GNU readline library for terminal - input management. To start Nim in interactive mode use the command - ``nim secret``. To quit use the ``quit()`` command. To determine whether an input - line is an incomplete statement to be continued these rules are used: - - 1. The line ends with ``[-+*/\\<>!\?\|%&$@~,;:=#^]\s*$`` (operator symbol followed by optional whitespace). - 2. The line starts with a space (indentation). - 3. The line is within a triple quoted string literal. However, the detection - does not work if the line contains more than one ``"""``. - - -Nim for embedded systems -======================== - -While the default Nim configuration is targeted for optimal performance on -modern PC hardware and operating systems with ample memory, it is very well -possible to run Nim code and a good part of the Nim standard libraries on small -embedded microprocessors with only a few kilobytes of memory. - -A good start is to use the ``any`` operating target together with the -``malloc`` memory allocator and the ``arc`` garbage collector. For example: - -``nim c --os:any --gc:arc -d:useMalloc [...] x.nim`` - -- ``--gc:arc`` will enable the reference counting memory management instead - of the default garbage collector. This enables Nim to use heap memory which - is required for strings and seqs, for example. - -- The ``--os:any`` target makes sure Nim does not depend on any specific - operating system primitives. Your platform should support only some basic - ANSI C library ``stdlib`` and ``stdio`` functions which should be available - on almost any platform. - -- The ``-d:useMalloc`` option configures Nim to use only the standard C memory - manage primitives ``malloc()``, ``free()``, ``realloc()``. - -If your platform does not provide these functions it should be trivial to -provide an implementation for them and link these to your program. - -For targets with very restricted memory, it might be beneficial to pass some -additional flags to both the Nim compiler and the C compiler and/or linker -to optimize the build for size. For example, the following flags can be used -when targeting a gcc compiler: - -``--opt:size --passC:-flto --passL:-flto`` - -The ``--opt:size`` flag instructs Nim to optimize code generation for small -size (with the help of the C compiler), the ``flto`` flags enable link-time -optimization in the compiler and linker. - -Check the `Cross compilation` section for instructions how to compile the -program for your target. - -Nim for realtime systems -======================== - -See the documentation of Nim's soft realtime `GC <gc.html>`_ for further -information. - - -Signal handling in Nim -====================== - -The Nim programming language has no concept of Posix's signal handling -mechanisms. However, the standard library offers some rudimentary support -for signal handling, in particular, segmentation faults are turned into -fatal errors that produce a stack trace. This can be disabled with the -``-d:noSignalHandler`` switch. - - -Optimizing for Nim -================== - -Nim has no separate optimizer, but the C code that is produced is very -efficient. Most C compilers have excellent optimizers, so usually it is -not needed to optimize one's code. Nim has been designed to encourage -efficient code: The most readable code in Nim is often the most efficient -too. - -However, sometimes one has to optimize. Do it in the following order: - -1. switch off the embedded debugger (it is **slow**!) -2. turn on the optimizer and turn off runtime checks -3. profile your code to find where the bottlenecks are -4. try to find a better algorithm -5. do low-level optimizations - -This section can only help you with the last item. - - -Optimizing string handling --------------------------- - -String assignments are sometimes expensive in Nim: They are required to -copy the whole string. However, the compiler is often smart enough to not copy -strings. Due to the argument passing semantics, strings are never copied when -passed to subroutines. The compiler does not copy strings that are a result from -a procedure call, because the callee returns a new string anyway. -Thus it is efficient to do: - -.. code-block:: Nim - var s = procA() # assignment will not copy the string; procA allocates a new - # string already - -However it is not efficient to do: - -.. code-block:: Nim - var s = varA # assignment has to copy the whole string into a new buffer! - -For ``let`` symbols a copy is not always necessary: - -.. code-block:: Nim - let s = varA # may only copy a pointer if it safe to do so - - -If you know what you're doing, you can also mark single string (or sequence) -objects as `shallow`:idx:\: - -.. code-block:: Nim - var s = "abc" - shallow(s) # mark 's' as shallow string - var x = s # now might not copy the string! - -Usage of ``shallow`` is always safe once you know the string won't be modified -anymore, similar to Ruby's `freeze`:idx:. - - -The compiler optimizes string case statements: A hashing scheme is used for them -if several different string constants are used. So code like this is reasonably -efficient: - -.. code-block:: Nim - case normalize(k.key) - of "name": c.name = v - of "displayname": c.displayName = v - of "version": c.version = v - of "os": c.oses = split(v, {';'}) - of "cpu": c.cpus = split(v, {';'}) - of "authors": c.authors = split(v, {';'}) - of "description": c.description = v - of "app": - case normalize(v) - of "console": c.app = appConsole - of "gui": c.app = appGUI - else: quit(errorStr(p, "expected: console or gui")) - of "license": c.license = UnixToNativePath(k.value) - else: quit(errorStr(p, "unknown variable: " & k.key)) diff --git a/doc/nimdoc.cls b/doc/nimdoc.cls new file mode 100644 index 000000000..37039f130 --- /dev/null +++ b/doc/nimdoc.cls @@ -0,0 +1,196 @@ +\ProvidesClass{nimdoc}[2022/04/17, 2018/01/01 LaTeX2e nonstandard class] + +\LoadClass[a4paper,11pt]{article} + +\usepackage[a4paper,xetex,left=3cm,right=3cm,top=1.5cm,bottom=2cm]{geometry} + +% for 2-sided printing with larger inner "binding" margin +%\usepackage[a4paper,xetex,twoside,left=4cm,right=2cm,top=1.5cm,bottom=2cm]{geometry} +% for e-readers with 1.77:1 aspect ratio (e.g. 1920x1080) +%\usepackage[xetex,paperheight=27.6cm,paperwidth=15.5cm,left=3mm,right=3mm,top=3mm,bottom=3mm]{geometry} +% for e-readers with 1.45:1 aspect ratio (e.g. 1200x825) +%\usepackage[xetex,paperheight=22.5cm,paperwidth=15.5cm,left=3mm,right=3mm,top=3mm,bottom=3mm]{geometry} +% for e-readers with 1.33:1 aspect ratio (e.g. 1872x1404) +%\usepackage[xetex,paperheight=20.7cm,paperwidth=15.5cm,left=3mm,right=3mm,top=3mm,bottom=3mm]{geometry} + +\usepackage{fontspec} +% logic to select default font with some fall-back fonts. +\IfFontExistsTF{Times New Roman}{% + \setmainfont{Times New Roman} % the default font + \typeout{========================================= nim: using Times New Roman} +}{ + \IfFontExistsTF{FreeSerif}{% + \setmainfont{FreeSerif} % fallback #1 - official GNU font, resembles Times + \typeout{========================================= nim: using FreeSerif} + }{ + \IfFontExistsTF{DejaVuSerif}{% + \setmainfont{DejaVuSerif} % fallback #2 - very widespread free font + \typeout{========================================= nim: using DejaVuSerif} + }{ + \typeout{!!!!!!!!!!!!!!!!!!! Fonts not found !!!!!!!!!!!!!!!!!!!!!!!} + } + } +} + +% default monospace font for code: +\usepackage{GoMono} +\usepackage{relsize} +% make this monospace font 2 steps smaller to hold 80-character line +\newcommand{\rstverbblockfont}{\smaller[2]} +\newcommand{\rstverbinlinefont}{\smaller} + +\usepackage{parskip} % paragraphs delimited by vertical space, no indent +\usepackage{graphicx} + +\usepackage{makeidx} +\newcommand{\nimindexterm}[2]{#2\index{#2}\label{#1}} +\makeindex + +\usepackage{dingbat} % for \carriagereturn, etc +\usepackage{fvextra} % for code blocks (works better than original fancyvrb) +\fvset{ + breaklines, + breakafter={=}:|\_\{\}[](){,}.;+-*/'", + breaksymbolleft=\color{red}{\ensuremath{\hookrightarrow}}, + breaksymbolright=\color{red}{\small\carriagereturn} +} +\fvinlineset{% + breaklines, + breakafter={=}:|\_\{\}[](){,}.;+-*/'", + % that does not work at all when we underline inline code by ulem :-( + commandchars=\\\{\} +} + +\usepackage{scrextend} % for the `addmargin` environment + +\usepackage[table]{xcolor} +\usepackage[urlbordercolor=blue,linkbordercolor=cyan, + pdfborderstyle={/S/U/W 1}]{hyperref} +\usepackage{enumitem} % for option list, enumList, and rstfootnote + +\usepackage[most]{tcolorbox} % boxes around admonitions, code blocks, doc.item + +\newtcolorbox{rstadmonition}[1][]{blanker, breakable, + left=3mm, right=0mm, top=1mm, bottom=1mm, + before upper=\indent, parbox=false, #1} + +\newtcolorbox{rstquote}[1][]{blanker, breakable, + left=3mm, right=3mm, top=1mm, bottom=1mm, + parbox=false, + borderline west={0.3em}{0pt}{lightgray}, + borderline north={0.05em}{0pt}{lightgray}, + borderline east={0.05em}{0pt}{lightgray}, + borderline south={0.05em}{0pt}{lightgray}} + +\definecolor{rstframecolor}{rgb}{0.85, 0.8, 0.6} + +\usepackage{booktabs} +\belowrulesep=0ex +\aboverulesep=0ex +\renewcommand{\arraystretch}{1.1} + +\newtcolorbox{rstprebox}[1][]{blanker, breakable, + left=3mm, right=3mm, top=1mm, bottom=1mm, + borderline ={0.1em}{0pt}{rstframecolor}, + before upper=\indent, parbox=false, #1} + +\newenvironment{rstpre}{% +\VerbatimEnvironment\begingroup\begin{rstprebox}% +\begin{Verbatim}[fontsize=\rstverbblockfont , commandchars=\\\{\}]}% +{\end{Verbatim}\end{rstprebox}\endgroup} + +\newtcolorbox{rstdocitembox}[1][]{blanker, breakable, + left=3mm, right=3mm, top=1mm, bottom=1mm, + borderline ={1pt}{0pt}{cyan}, + before upper=\indent, parbox=false, #1} + +% Inline code formatting: grey underline, +% use \Verb from fvextras e.g. to display -- correctly as double - +\usepackage[normalem]{ulem} +\newcommand\rstuline{\bgroup\markoverwith{\textcolor{rstframecolor}{\rule[-0.8ex]{2pt}{1.0pt}}}\ULon} + +\newcommand{\rstcode}[1]{% +{\rstverbinlinefont\Verb{\rstuline{#1}}}% +} + +\newcommand{\rstcodeitem}[1]{\Verb{#1}} + +\newenvironment{rstdocitem}{% +\VerbatimEnvironment\begingroup\begin{rstdocitembox}% +\begin{Verbatim}[fontsize=\rstverbblockfont , commandchars=\\\{\}]}% +{\end{Verbatim}\end{rstdocitembox}\endgroup} + + +\newenvironment{rstfootnote}{\begin{description}[labelindent=1em,leftmargin=1em,labelwidth=2.6em]}{\end{description}} +\ifdim\linewidth<30em + \def\rstoptleftmargin{0.4\linewidth} + \def\rstoptlabelwidth{0.35\linewidth} +\else + \def\rstoptleftmargin{12em} + \def\rstoptlabelwidth{10.5em} +\fi +\newenvironment{rstoptlist}{% +\begin{description}[font=\sffamily\bfseries,style=nextline,leftmargin=\rstoptleftmargin,labelwidth=\rstoptlabelwidth]}{\end{description}} + +\usepackage{multirow} +\usepackage{tabulary} % tables with adjustable cell width and no overflow +% make tabulary prevent overflows (https://tex.stackexchange.com/a/195088) +\tymin=60pt +\tymax=\maxdimen +% to pack tabulary into a new environment, special syntax is needed :-( +\newenvironment{rsttab}[1]{\tabulary{\linewidth}{#1}}{\endtabulary} + +\newcommand{\rstsub}[1]{\raisebox{-0.5ex}{\scriptsize{#1}}} +\newcommand{\rstsup}[1]{\raisebox{0.5ex}{\scriptsize{#1}}} + +\newcommand{\rsthA}[2][]{\section[#1]{#2}} +\newcommand{\rsthB}[2][]{\subsection[#1]{#2}} +\newcommand{\rsthC}[2][]{\subsubsection[#1]{#2}} +\newcommand{\rsthD}[2][]{\paragraph[#1]{#2}} +\newcommand{\rsthE}[2][]{\paragraph[#1]{#2}} + +\newcommand{\rstovA}[2][]{\section*[#1]{#2}} +\newcommand{\rstovB}[2][]{\subsection*[#1]{#2}} +\newcommand{\rstovC}[2][]{\subsubsection*[#1]{#2}} +\newcommand{\rstovD}[2][]{\paragraph*[#1]{#2}} +\newcommand{\rstovE}[2][]{\paragraph*[#1]{#2}} + +% Syntax highlighting: +\newcommand{\spanDecNumber}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanBinNumber}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanHexNumber}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanOctNumber}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanFloatNumber}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanIdentifier}[1]{#1} +\newcommand{\spanKeyword}[1]{\textbf{#1}} +\newcommand{\spanStringLit}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanLongStringLit}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanCharLit}[1]{#1} +\newcommand{\spanEscapeSequence}[1]{#1} +\newcommand{\spanOperator}[1]{\textbf{#1}} +\newcommand{\spanPunctuation}[1]{#1} +\newcommand{\spanComment}[1]{\emph{#1}} +\newcommand{\spanLongComment}[1]{\emph{#1}} +\newcommand{\spanRegularExpression}[1]{#1} +\newcommand{\spanTagStart}[1]{#1} +\newcommand{\spanTagEnd}[1]{#1} +\newcommand{\spanKey}[1]{#1} +\newcommand{\spanValue}[1]{#1} +\newcommand{\spanRawData}[1]{\textbf{\textcolor{darkgray}{#1}}} +\newcommand{\spanAssembler}[1]{#1} +\newcommand{\spanPreprocessor}[1]{#1} +\newcommand{\spanDirective}[1]{#1} +\newcommand{\spanCommand}[1]{#1} +\newcommand{\spanRule}[1]{#1} +\newcommand{\spanHyperlink}[1]{#1} +\newcommand{\spanLabel}[1]{#1} +\newcommand{\spanReference}[1]{#1} +\newcommand{\spanOther}[1]{#1} +\newcommand{\spantok}[1]{\fbox{#1}} +\newcommand{\spanPrompt}[1]{\textcolor{red}{\textbf{#1}}} +\newcommand{\spanProgramOutput}[1]{\textcolor{darkgray}{\textbf{#1}}} +\newcommand{\spanprogram}[1]{\textbf{\underline{#1}}} +\newcommand{\spanoption}[1]{\textbf{\textcolor{darkgray}{#1}}} + +% Never allow text overflow to margin: +\setlength\emergencystretch{\hsize}\hbadness=10000 diff --git a/doc/nimdoc.css b/doc/nimdoc.css index 6025d6601..0c399e4c1 100644 --- a/doc/nimdoc.css +++ b/doc/nimdoc.css @@ -14,6 +14,9 @@ Modified by Boyd Greenfield and narimiran --primary-background: #fff; --secondary-background: ghostwhite; --third-background: #e8e8e8; + --info-background: #50c050; + --warning-background: #c0a000; + --error-background: #e04040; --border: #dde; --text: #222; --anchor: #07b; @@ -32,13 +35,22 @@ Modified by Boyd Greenfield and narimiran --escapeSequence: #c4891b; --number: #252dbe; --literal: #a4255b; + --program: #6060c0; + --option: #508000; --raw-data: #a4255b; + + --clipboard-image-normal: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: black' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E %3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2' /%3E %3C/svg%3E"); + --clipboard-image-selected: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: black' viewBox='0 0 20 20' fill='currentColor'%3E %3Cpath d='M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z' /%3E %3Cpath d='M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z' /%3E %3C/svg%3E"); + --clipboard-image: var(--clipboard-image-normal) } [data-theme="dark"] { --primary-background: #171921; --secondary-background: #1e202a; --third-background: #2b2e3b; + --info-background: #008000; + --warning-background: #807000; + --error-background: #c03000; --border: #0e1014; --text: #fff; --anchor: #8be9fd; @@ -57,66 +69,54 @@ Modified by Boyd Greenfield and narimiran --escapeSequence: #bd93f9; --number: #bd93f9; --literal: #f1fa8c; + --program: #9090c0; + --option: #90b010; --raw-data: #8be9fd; -} - -.theme-switch-wrapper { - display: flex; - align-items: center; -} - -.theme-switch-wrapper em { - margin-left: 10px; - font-size: 1rem; -} - -.theme-switch { - display: inline-block; - height: 22px; - position: relative; - width: 50px; -} - -.theme-switch input { - display: none; -} - -.slider { - background-color: #ccc; - bottom: 0; - cursor: pointer; - left: 0; - position: absolute; - right: 0; - top: 0; - transition: .4s; -} -.slider:before { - background-color: #fff; - bottom: 4px; - content: ""; - height: 13px; - left: 4px; - position: absolute; - transition: .4s; - width: 13px; + --clipboard-image-normal: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: lightgray' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E %3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2' /%3E %3C/svg%3E"); + --clipboard-image-selected: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: lightgray' viewBox='0 0 20 20' fill='currentColor'%3E %3Cpath d='M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z' /%3E %3Cpath d='M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z' /%3E %3C/svg%3E"); + --clipboard-image: var(--clipboard-image-normal); } -input:checked + .slider { - background-color: #66bb6a; -} - -input:checked + .slider:before { - transform: translateX(26px); -} - -.slider.round { - border-radius: 17px; +@media (prefers-color-scheme: dark) { + [data-theme="auto"] { + --primary-background: #171921; + --secondary-background: #1e202a; + --third-background: #2b2e3b; + --info-background: #008000; + --warning-background: #807000; + --error-background: #c03000; + --border: #0e1014; + --text: #fff; + --anchor: #8be9fd; + --anchor-focus: #8be9fd; + --input-focus: #8be9fd; + --strong: #bd93f9; + --hint: #7A7C85; + --nim-sprite-base64: url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARMAAABMCAYAAABOBlMuAAAACXBIWXMAAAsTAAALEwEAmpwYAAAFFmlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4gPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRvYmUgWE1QIENvcmUgNS42LWMxNDggNzkuMTY0MDM2LCAyMDE5LzA4LzEzLTAxOjA2OjU3ICAgICAgICAiPiA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPiA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6cGhvdG9zaG9wPSJodHRwOi8vbnMuYWRvYmUuY29tL3Bob3Rvc2hvcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RFdnQ9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZUV2ZW50IyIgeG1wOkNyZWF0b3JUb29sPSJBZG9iZSBQaG90b3Nob3AgMjEuMCAoV2luZG93cykiIHhtcDpDcmVhdGVEYXRlPSIyMDE5LTEyLTAzVDAxOjE4OjIyKzAxOjAwIiB4bXA6TW9kaWZ5RGF0ZT0iMjAxOS0xMi0wM1QwMToyMDoxMCswMTowMCIgeG1wOk1ldGFkYXRhRGF0ZT0iMjAxOS0xMi0wM1QwMToyMDoxMCswMTowMCIgZGM6Zm9ybWF0PSJpbWFnZS9wbmciIHBob3Rvc2hvcDpDb2xvck1vZGU9IjMiIHBob3Rvc2hvcDpJQ0NQcm9maWxlPSJzUkdCIElFQzYxOTY2LTIuMSIgeG1wTU06SW5zdGFuY2VJRD0ieG1wLmlpZDplZGViMzU3MC1iNmZjLWQyNDQtYTExZi0yMjc5YmY4NDNhYTAiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6ZWRlYjM1NzAtYjZmYy1kMjQ0LWExMWYtMjI3OWJmODQzYWEwIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9InhtcC5kaWQ6ZWRlYjM1NzAtYjZmYy1kMjQ0LWExMWYtMjI3OWJmODQzYWEwIj4gPHhtcE1NOkhpc3Rvcnk+IDxyZGY6U2VxPiA8cmRmOmxpIHN0RXZ0OmFjdGlvbj0iY3JlYXRlZCIgc3RFdnQ6aW5zdGFuY2VJRD0ieG1wLmlpZDplZGViMzU3MC1iNmZjLWQyNDQtYTExZi0yMjc5YmY4NDNhYTAiIHN0RXZ0OndoZW49IjIwMTktMTItMDNUMDE6MTg6MjIrMDE6MDAiIHN0RXZ0OnNvZnR3YXJlQWdlbnQ9IkFkb2JlIFBob3Rvc2hvcCAyMS4wIChXaW5kb3dzKSIvPiA8L3JkZjpTZXE+IDwveG1wTU06SGlzdG9yeT4gPC9yZGY6RGVzY3JpcHRpb24+IDwvcmRmOlJERj4gPC94OnhtcG1ldGE+IDw/eHBhY2tldCBlbmQ9InIiPz4JZNR8AAAfG0lEQVR4nO2deViTZ7r/7yxkJaxJ2MK+GCBAMCwS1kgUFQSKK4XWWqsz1jpjp3b0tDP1V+eqU391fqfT/mpPPd20drTFDS0KFEVWJSGAEgLIZpAICBJACIRs549Rj1WILAkBfD/XlevySp68z/0S3+/7vPdzLyidTgcLkU2bd+z39/f/q1gshsrKSoJELFCa2iaEuU9K6kb+8uXxv54/fzE8L/eswNT2zCfQpjbAGKS8lPFKSEjIXiaTCSEhIeDj4xNnapsQ5j6rktZGp6UlfxIdzQVzCplmanvmG1hTG2BIAtlc26CgoDfT0tL2e3l5AQCAjY0NkMnk/a9s2k6rrKw8UV8n1JjYTIQ5RlAw14KzmL3xze1vfJyUuMJaq9UCFovFm9qu+YbBxcSPFUYkk8l2Q0NDsvo6ocrQx5+I8Ih4bz6f/0l8fHyKlZXV4/dRKBQwmcwwMpn8A4FAoPgHhH9bV1sxa488wZxoaycnJ/a9e/duCa5fkc3WvAiTI4Ib77p+XdqHG9anbfLy8gAAgLGxMdBpF+bjvzExqJj4scKI0dHRnwQHB++orq7+AgDeMuTxJ2Jl4rqU9PT0EwEBAUQCgTDuGAaDAampqYepVKpHUHDk325Ulw0a266YuFW+Gzdu/MDPz29jfn7+XgA4aOw5ESZP6kvpCXv3vnM8NiaSamVl+fj9BepGNDoGFRN7e/slcXFxO1xcXMDJyWnH7j//H/fi4uJdgutXmgw5z5O8smn7X9euXbvf29sbMBjMhONQKBRYWVlBbGzsbjMzM3JoOG+/sKKwy1h2rd/4elpGRsYuLy+vaDweD2w2Oy1h5ZrCvEunEaeeiVnMiabyl/F2/+X9P+8JDPQHHA5napMWBAYTk6DgSNuEhIS9DAYDAP7tq1i6dOkqOp3OWbNu0wens44emeoxA9lcWwKBYEMkEm2JRKIdHo+3QKFQWJ1Op8ZgMER3d/dVq1evTnFycpr0MSkUCsTExGzH4/Gk1LTME/39/TI0Go1FoVCg1WrVY2NjipGRkcGRkRH5dPwrEZHLXMPCwjJSUlIy3dzcfB+97+rqGhYSEpIOAIiYmBguN3zL77dt3uPh4W5qUxYUBhMTb2/vjeHh4cvR6P/dILK0tITIyEg7BweHr363/Z3Ampqaf1Zcu/zMKiVsyVJvMplsRyKR7IhEor2FhYUbhUJhJCYm2pFIJB6JRAIymQx4PB7QaDRoNBowMzMDJycnwOOn7icjEokQGxu7icFgbLp///7jFY1WqwWlUgkjIyOgUCgO7Ni5Rz48PCwfHh7uGRkZeaBQKOSjo6ODCoVCXlNVKn/6uCsT13FXrVr1emho6BYKhfLMnP7+/omrU9LPX8g+UThloxEMxqJFXjxESAyPQcSEExrLWLNmzW57e/txP/fw8ABHR8cdDAaDt3xF2ru9vb03sVgs0cbGxs/FxWVZUlISj0aj+dna2oKtrS1M5PcwJCgUCry8vODRrs84vPfoH6OjoyCXy6Gvr+/R6+CWrX9s7evrk/b19bWr1Wqli4sLZ8OGDe95eXmxUSjUuAd0cHDwjoqK2sYKXFIhvnldYYTTQpgU4/8+jyASCYDGoCd+ZkYYF8OICYezl8PhuOkbQyAQIDo62s/NzS2np6cHbGxsgEajAYFAAAwGA1gsFia6CE0NgUAABwcHsLe3B61WC2q1eo9WqwWNRgNKpRLUajUQiUSgUCh6zwGHwwGTydzo5+eXBQBnZu8MEJ5keHhYPqyYWMtHR0ZBpVIhYj9FUDONgOUvT12+du3avMDAQJjssdRqNWCxCyrEZdLodDoQi8Ulx44de628NL/V1Pa8iERE8l2dHB2CJvpcq9Nqbt1qKURWj1Njxld0ZGTkAW9v70kLCQC8sEIC8O/HKx8fn2gmk8kHgCk7pRFmzrWyAikASE1tx0Jj2uH0EZHL/N7YtuvT4OBgzmz4OBYSeDweIiMjt2S++vtMP1YYEmmJsCCY8mNOIJtr6+zsHBcZGXmIw+G4mZubG8m0hU9HRwcUFxe/KxQKTyDRsQjznSmJCS9+dVRERMTfQ0NDo2xtbfUGiSFMjtHRUaitrc3Jzc09kHvxVLmp7UFAmC6oZQkvrZLL5RJhReHtiQb5scKIXC7371FRUX90dnYGIpE4JR8Jgn40Gg20t7fXFxYWfnr9+vWjz8sdYi+Osh4vzgUBwZSgtu94V+fs7Hx7YGCgra6u7khLS0u2RCwYeTQgKmYFh8fj/f/g4OAldnZ2prR1wdPd3Q1CofBQSUnJkdLi3N8E93FCY6k+Pj48FxcXjlar1ZSWlh65VvYr4kREmDNg79+/D3FxcW5OTk5uXl5evNbW1tL0jK3ZXV1d1ykUintycvInoaGhdkj+gvGxs7MDPp+/m0AgWMQvS/lyeHhYTqPRPJycnIJSU1NZ3t7eW2g0Gly/fv2oWq1Gij0hzClQ/gHhpLS0tEM8Hm/7I8Ho7++HlpYWsLa2Bg8PDxOb+OKhUCigqakJ7t+/D25ubuDu7g4oFAp0Oh08ePAAvv7666TTWUdzTG0nAsKTYMU3ryuSU18+4+bmFrZo0SIOAICVlRUsXrx4zkakLnRIJBI8CgJ8MtdJp9NBZ2enqL29XWRC8xAQxgUNAHD+3L8KGhoaCp78ABES04JCoX4jJAAAAwMDUFtbe96YpRMQEKbL41DU5ubmko6Ojj2PSgggzD36+/vrb9y4cX425zzw93/8EBjon2is44+NjSkePBjqGRwc7G5v7xBV19w8U5B/3qgrr9+/uWtXUuKKD/TZ9MXh/066/OuFmunO8dGBQ98HBbGSp/t9U6LRaDXK0dHBoeFhuVzeL22/0yFqamopufjLqRJ933ssJi0tLSXV1dWHGAzGbuObOzs8ubqa71vZKpUKOjo6blwpOF8zm/Mu5cVkLlkSaswprAHAaVihgK7O7oSGxltvfXLon3nXK4RHT2cdN4pfKDCAlZyUuMJan02nTmczAaBmunPw4qI3cbnh0/36XICq0+lgcPABp7OrK629vUP5z8++LLh2XXD05L++yxrvC4/F5EZ12WBS8saLS5Ys2U2lUufUY45SqQSlUgkqlQrUavXj19jYGGg0GtBoNKDT6UCn05VotVq1TqfToFAojFar1eh0Og0Wi8XhcDgeGo1+/PhgZmYGOBwOsFgsmJmZ/eY1F+nt7YXa2trs2Z73wdCQBgCMHp1IJpHA09MdPD3dLRIS+OtKisvWvbP7vf2lZdePVFwzbHTwyMiI3hidkZFRUKvUYzOZ48HQkBIA5nWqBAqFAktLC7C0tADmIh88Pz4uMSyUk7hn776DV4tKPn/6d/lNxp1MJqsRCASf8vn8XdMpOjRTVCoVjI2NgUqlAq1WCyMjI9DX1wf379+Hvr6+/Q8ePOgdGRmRKxSKx0WLFAqFXKlUKnQ6nUar1arHq47mxwrD4/F4Eg6HI2GxWDwej7cgkUjWFAqFam5uTjU3N6eRyeQPLSwswNraGqysrIBAIDwWFywW+zja11Qi29LSclIikeSZZPJZBovBAI8XA8HBQR9kZZ3lR8cmvFZSlGe00p8IkwONRkNERBj4+i7a4+XpHv307/IbMakWlciXJbx0nMPh7Jqo0JGh0el0MDo6Cl1dXSCVSkEmk7177969W319fe1DQ0M9KpVKoVarlWq1WjndNhUPG3ApAWDcOxLTLwSDwWAOotFoDBaLxRMIBAsrKysne3t7Xzqd7k2n0/c4OzsDlUoFHA4364IyMDAATU1NxdWikhcq6tXKyhJezljPJZKI2eERS5cZeoWCMD2srCwhPX0tVzk2djiCG//GtfLLUoBxShB0dHTU3Lx580sLC4vtJBLJKMZoNBqQSqUglUqPdnR01PT09DT19/fLHjx40DM0NNQ72933GiSVGgB4JFQK+LfoSAGgnL04yppEIh2xtLS0t7GxcaFSqR7Ozs4fMRgMcHR0nJX8pJs3b54Ui8UXjT7RHIRMIkFK8irfwcEHPwQELUmqvYHUGJkLmJubw8YNa/i9vfffY/px3myQiDTPiEl9nVDDX576jaenZ7SnpyfLUJNrNBqQyWRw+/bt4x0dHTdkMlltV1dXw/XygjkdEv4wB0YOAK0AUM70C8HQ6fSzdDrdm0qlejg6OrLc3Ny2MBiMadWjfR4PHjyAmzdvZs/1v5MxoVAokJK8iicWS95k+nH+s0EiQhqpzQGoVFtYk5a87ba0XQAA34xbpagg/5zoT7s/OGNnZ8eaaYkBuVwOnZ2d5VKpVNTS0lLS2NhYWFVZ3Dujg5qQh6uY+ocvCAiKIPn4+Jz19PSMdnV15VCpVL6Dg4NBViw6nQ5EItHRpqamqzM+2DzHzo4O69amftLQeKsAZrDLgmBY/PyYsCIhfs+SiKUFE5Y8EwqFx11cXDihoaFTjjFAoVAwPDwMHR0dourq6jNCofDHhZqUVnvjmgIAcgAgJyg40mLRokX8kJCQjT4+PussLS1n1JPl7t27UFxcfHguB6mNjY2B7G4naNRTWyygUCjAYDGAx+PB0sICSCSi3vFYLBbCwjjA8vddBQtATKb7d3saBwc7IJPJBpsHjUGDGRYLJBIJLK0sAfucmyIGg4FFi3y8AwNZtycUk5KiS02vvf7WWQaDkejg4DApQwAeh3xDaWnpPoFAcPxFqnP6sEvgGf+A8Bx3d/cvIyIiNi1evHjT8wpNj8fAwACUlZW9P9dD5+/ckcFbf9gd2dcnn9LNAovF4inmZHtXNxdOdBR3+/JlS33pdP29wolEInA4weuiYxOy5vvuTkeHDHb+8c8xvb33Z3R9/N+Df+uIjYk02DwkEsna2trS1d/fNyGeF7uTyw1/7g3R3t4O2OxA/TVghULhcQqFQk1JSfmYSNR/5wD4d6EfgUBwvLS09IhUKhW9qAV5H9YjKQwJi6uvrKw8ERoamhkSEpKp7w7yJEqlEiQSyZmysrJv53qjdaVSCZdyTk+3qFMrAJRHRPLPN95qeifj5fU7mYt8JhyMRqMhMJDFdnF25gDAvBYTpXIMWlpay2fq/8m5mDcIABYGnEcGAGI/VlhBZWX1yZdSkz55OX0dV5+7w9bGGvz8mPrFpK62QskJjf2GTqd7x8bGbpnID4BCoUAmk0lLSkqOiESik2UleS/MakQflYKrXQDQxY1a3tTe3i6KiIjY5OXlxX7e9+rr6wsuXbr0t4ffn9OgMWjghMZQRcLp+8GulRVI/QPC37Wxtnal0ajJtjY2E451ZjiBra31vE9lR2PQQKFQaAAwo98Yi8Xq9fpPd56HO6rlvKWJv/PwcK+JilyCmajWMw6HAzs7+rMFpQOCIn6zHywSFvXm5eUdFAqFZ9Rq9bgHa2trq79w4cK+zz49cAARkmcpL81v/a/Dhz49d+7c3qqqqjyVSjXuOJ1OBxKJpDw3N/fA5V+zax6978cKw/sHhM/raMrnUVdboSy4fPWQSFSjd5yFBQWIRNKEd2IEw1J4JUd88WL+R51d3XrHWVDMnxUTa2tr1zXrNiUGsrmPf7DS4tymCxcu7Kuurs55+kKQSqVN586d23vs+8NHDXUCC5Wzp3/Iy8rKeruysvLM2Nhvo7VVKhXU1tYWnj17du/T7UOdnZ2D7OzsfGGB09raVi4S1RzXl0eFw+EAj8chYjKLVFffyOrq1C8mJBLpWTFRKBRyDofzC4vFWvXk+1ev/CLOzs7eKxAIslQqFeh0Oujp6enKzs7em/XTd7OayTqfKb56sT4rK+sPAoHg5KO/o0KhAKFQmHXy5MkdF3/5+TeZmctXpIXZ29v7zqVcKWNRX1epuXu3U/y8pEw0GmndOZt0dnXVDw0P6/W5oNHoZ30mQ0NDPb29vfvj4+Pf3rR5B/7od188XnEUXr4gDgmL+0NfX5/U19d3d3l5+YGfTnyDtLmcIhXXLsu4UcvfR6PRGGtra9eysrIjYrE45+kt4Fheou/69es/unnz5vm7d+/Wmsre2WRkZGTQ1DYg/JYGiUiTm1ugBAC9IfHPiEmDpFITE7fqJI/H27lmzZpDq5LWtz55t6wUXO3ihMYerK+vz2tpaUFaM0yT8tL81ujYle+TSCTrvEunBU9/voTLd92wYcPHVCqV39XVdXCu7+oYCp1O90Kc50Jk3I5+xVcv1jc3N5d4enpSMzIyvkpK3sh78nORsKg3++yPBS/q1q+hKCm61DSekERGJ3ikp6d/ERsbm1xVVXWwtbX1hRFtFAqFPMLMUyZsDyoQCI7LZDKIiIjwzczM/GpV0vro2TTsRSUqZoX3+vXrP1u9enXi0NAQiESirIdRtggIc5oJ40zq6uryGhoa8ry8vBJCQ0O9USjU94mrN7yWc+EnvaXb5gJMvxCMp6cnl0Kh2Le1tZVXXLs8L1LXefGrWRkZGZ/x+XyeUqkEkUh0vqenZ14HZyG8OEwoJjdrygd37NxTEBkZmWBtbQ3BwcEeKBTq+/UbX3/355Pfzlmn66qk9dGbN29+k8PhbCSRSNDZ2Snb9ae/HCkpKTksEhbN2QTD5NSX+Vu3bj0cHBzsjcFg4O7du1BWVvbNwxB9BIQ5j94I2Fu3bhXW19cDl8sFLBYLHA7Hg0wmf/e77e84ffXlPz6fLSMnQ2paZkJ4eHjmtm3b+B4eHvZkMhlQKBTY29s72dvbfxgUFJT8x7ffP1NRUfHjXErnZ/qFYKKjo7dt3rz5g8DAQPtH/XHa2tpqGhsbC55/BASEuYFeMblz505NTU3NgfDw8PcwGAygUCjw9fW1IJPJn/1130Hv0tLSI4WXL4hny9inYS+Osvbz80tgMpn8jIwMPovFch2vpoiDgwM4ODhwfH19OYsWLeJv3/Hu+cbGxquzXZz5aZYlvMRJT0/fFhkZue3JZmfd3d0gEolOIr4ShPmEXjFpkFRqXlrzSnFnZ+d7Tk5OjzNfXVxcICMjY6ezszNnVdL6vU8HWhmbgKAIkrOzMyc1NTXz0YU4maAuOp0OK1as4EVFRfGEQqHg1dfePHzr1q2rs71S8WOF4f38/BLS09M/iIyM5DxdxLq5uVlcVVU1bgVwBIS5il4xAQCQyWRigUBwJikpKe3JVGQcDgdLly7l2tranti0ecf7IpEoy9hbxX6sMDydTvdevXr1ltjY2F3u7u6AxT73FJ7B3Nwc4uLiwthsdphQKCzZkL7l0/r6+oKbNeVG90+EhMXZL1++fFtycvKHrq6uz4igUqmE5ubmEiTHCWG+8dwrUXD9imz9xtd/jIuLS7N5KpsTjUZDUFCQE4PB+F4oFGYmJW888Mv5k4UTHGpGxC9LYaenp78VEhKyxdHRESgUyoyOh0KhwNraGuLi4qIDAgKi6+rqyjekb/mHMSN6N6RvSdu+ffseNpsdZm09ftuW+vp6EIvFSB9hhHnHpG7rUqm0orW1tdXS0tLj6TIEaDQaaDQaxMfH811dXTl/3Xfw+JUrVz411J01cfWG6IiIiC07d+5McHNzs7ewMGyOFw6HAwcHB6BSqVx3d/fwz7/4rkAgEBwXCoUnHpZonDGrU9J5MTEx27du3Zrm4uKC0beaqq6u/ry+vj7XEPMiIMwmkxKTimuXZe/u+fCkp6fnexPdUfF4PPj7+1szGIydLi4unF1/+kvenTt3RG1tbRXTqfma8lIG39/fP/HVV19NZrFYHpMpzjQTzMzMwNPTE+Pp6Zng6emZ4Ofnl5CesfV8bW1tznQe3/wDwvFeXl7Rvr6+Ca+88kpaUFCQh74GXzqdDrq7u6GpqankRQmdR1hYTNrhUFVVlcXj8d6ysrKy0OfstLS0hPj4eC6Xy+U2NzeDRCI5/sa2XeX37t1rGhwc7BoYGJBN1P+FFbiE5OzszGaxWImvvvrqpoCAAKfp+ERmCpPJBCaTmcnhcDJLS0u/TE59+YxUKhXoi/lg+oVgrKysGJaWlna2trYeaWlpXDabvTMgIGDSfp2KiorzbW1tL0zoPMLCYtJX6uVfs2u++PKowMPDgz+ZIslEIhECAgKAxWJlajSazJ6eHmhra4PW1tZvtmz9o6Czs7O+r6+vfWxsbFir1WosLCzsV6xYkcnj8d7z9vaelmPV0Hh5eYGnp+f2mJiY7UVFRZ/HL0v5tru7+5ZGo1FisVg8Docj4fF4CxsbG1c+nx/m7e39sYeHB7i4uIC5ufmU6r4ODQ1BZWXlifkSrYuA8DRTumIrKytPent78728vCb9HRQKBVgsFhwcHIBOpwObzd4yNja2RaVSwdDQEHR1dcHo6CjQaDRwdXWdsWPV0KBQKPDw8AA7O7udERERO2tra2FgYACoVCo4OTkBjUYDMpkMeDz+8WuqaLVaaGxsbL19+/YzSX8ICPOFqYrJidDQ0AwvLy/e80c/CwaDARKJBI86BdJoNHB3dwe1Wj0nViL6IJPJwGQywdnZGZRKJRAIBDBUx8OBgQEoLS39BtkORpjPTJg1PB61N64pmpqarvb39xvUiLkuJE9CJpPBxsbGYEICANDZ2SlHgtQQ5jtTEhMAgLq6ulyJRFJvDGNeREZGRkAikRSUFuci2cEI85opi0l+7hmBWCzOeV6dToTJcfv27cHr168jxbgR5j1TFhMAgObm5hKZDNl0MAQtLS3Xzpw6hkS8Isx7piUmUqlUIBAIJuyjgzA5Ojs7QSKRINGuCAuCaYmJsKKw68qVK59KJJIu5HFneiiVSigqKjouEolOmtoWBARDMC0xAQC4+MvPJadOnXq3ra1N8yL0dDEkOp0OSktLy/Pz8w8+3d4CAWG+Mm0xAQA4fuy/jl+8ePGju3fvGsqeBY9Wq4XKysrWU6dOvX31yi8mKyyFgGBoZiQmAAD/79D+fadPn96PCMrz0el0UFVV1frtt9+mj9fiAgFhPjNjMQEAyMvLO3Ds2LE/tLS0INmuerh27Vr9999//xoiJAgLEYOEntbVVigB4PNNm3cMpqSkfMRms50McdyFgkqlgqKiovJTp069nZ97BhEShAWJQePYj373xdF1GzbLFQrFx6Ghob766ne8KNy7dw+KiopO5ubmfmTK4tsICMbG4EkxWT99d35l4rre/v7+D0NCQvh0Ot3QU8wL1Go1SKVSTX5+/sH8/PyDSP8bhIWOUTLsLuVklQcFR65pbGzcvnLlyvfc3NwsCASCMaaac+h0OhgaGoLq6uqaCxcu/OV01tGcTw7uM7VZCAhGx2jpug/vxAd58atzoqKitq1cuXKnvb29saabE+h0Oqiurpbm5eUdrK6uPlspuDrvY0hmO4YIhUIBGq1/X2CmNqFQKL3/79HomZ/z82xEowyy9zFr80zGDqPn/hdeviBmL47ad+fOnRsRERGbQkNDo62srIw97azT2dkJxcXFx0tKSo7Mdh8hY4LD4TDPH2U4MFjMc6tLmZmZzaj+Aw6H0/t9PB4PGCxmRudNJBL0ngeZTAI0Gj3jv+1szfM88Hic8cUEAKCmqlQOAN/ELU2qkEgkySwWK3HRokVcBoMxG9MbDZ1OB83NzdDU1FRQW1t7XiAQHJ+ovu18pbr6Rg6L5ZtoM0EhcUPT0tJW8tWRb0vQqIkvgKqqmhnVfrl2TfANXo+gjKlUio4OWc1M5sjOzjnQUH8rbqLPu3t6moaGhmfc+3q25tGHUqmECoEIUKbIrVkcEkONiIh4jcvlvu7s7OxLo9GmVe7QVCgUCujq6oKGhoaCioqKo9XV1WeM3YDMVPDik1gpyas+XrVyeaKXl8czjyANjbcgI/MNmkg49Q4ECPOH3NyC4RUr+M8IcHt7B1y9WlKRl3/5kElKnD1sfXEoJCzueEBAQGJYWFgGk8nk2djYAIFAgLm4pTw6Ogqjo6Mgl8vhxo0b50tLS4/U19fnLvS2FIWXfxEDQNLmLW9ueW1TxtchHDaQyWRTm4VgYkZHR6G+vhF+/NfP+y5e+vVjiVgwZpKVydOwF0dZW1lZOTGZTD6bzU4LCAiIptPp8HTDL1MwOjoKLS0tUFdXd1IsFudIpdKKgYGB7tloJTrX4MUnsVJTEj9etzY10dHRAQAAGm81wcsZW5CVyQInL69gNCGBjwcAGBx8ANnncypOnTr3H9nn/reD55wovvrQpyIHAHFUzIocGo3mQaPRfBwdHVlubm7bXF1dgcFgABqNNvruglwuh7t374JMJoOOjo7P79y5I+ru7m7q7e1tXQi7MzOh8PIv4pCw2DdaWtte37Au7aPIyCWAxWABjUbPif9HCMbjURtKiaQBfvr5zH9evlJ0uLQ4r/nJMXNiZTIRrMAlJAcHB18HBweWo6Mjy8rKajeJRAJLS0uwtLQECwsLoFAogMfjAYvFgpmZ2XNXMyqVCoaHh2FoaAiGh4cfvwYGBqCvrw+6u7vfvnfvXlNvb29rT09Pq0QsUM7S6c4rNqS/lrZ5U+YPRBKR9M7u9xwqBUUvtNAudH766XSLE8PR49ixE78/8tVnX403Zk7fUR46NUUAIPIPCMdTKJTdNjY2QKPRgE6nA51OB1tbWyCRSIDD4YBAIAAejwcCgfDYUajVakGlUoFarQadTvfY79HX1wf9/f0gl8tBLpfDvXv3HvXw+dxQPYYXMj+d+P7Mmzv+5OHr6/OJWq1GBHeB09TcUiKuq/coKS3/eqIx/wPkiIXC3w6YjAAAAABJRU5ErkJggg=="); + + --keyword: #ff79c6; + --identifier: #f8f8f2; + --comment: #6272a4; + --operator: #ff79c6; + --punctuation: #f8f8f2; + --other: #f8f8f2; + --escapeSequence: #bd93f9; + --number: #bd93f9; + --literal: #f1fa8c; + --program: #9090c0; + --option: #90b010; + --raw-data: #8be9fd; + + --clipboard-image-normal: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: lightgray' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E %3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2' /%3E %3C/svg%3E"); + --clipboard-image-selected: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' style='color: lightgray' viewBox='0 0 20 20' fill='currentColor'%3E %3Cpath d='M8 3a1 1 0 011-1h2a1 1 0 110 2H9a1 1 0 01-1-1z' /%3E %3Cpath d='M6 3a2 2 0 00-2 2v11a2 2 0 002 2h8a2 2 0 002-2V5a2 2 0 00-2-2 3 3 0 01-3 3H9a3 3 0 01-3-3z' /%3E %3C/svg%3E"); + --clipboard-image: var(--clipboard-image-normal); + } } -.slider.round:before { - border-radius: 50%; +.theme-select-wrapper { + display: flex; + align-items: center; } html { @@ -141,23 +141,39 @@ body { padding: 0; box-sizing: border-box; } -.column, -.columns { +.column, .columns { width: 100%; float: left; box-sizing: border-box; - margin-left: 1%; + margin-left: 1%; } + +@media print { + #global-links, .link-seesrc, .theme-switch-wrapper, #searchInputDiv, .search-groupby { + display:none; + } + .columns { + width:100% !important; + } } -.column:first-child, -.columns:first-child { +.column:first-child, .columns:first-child { margin-left: 0; } +.container .row { + display: flex; } + .three.columns { - width: 19%; } + width: 25.0%; + height: 100vh; + position: sticky; + top: 0px; + overflow-y: auto; + padding: 2px; +} .nine.columns { - width: 80.0%; } + width: 75.0%; + padding-left: 1.5em; } .twelve.columns { width: 100%; @@ -223,6 +239,12 @@ select:focus { } /* Docgen styles */ + +:target { + border: 2px solid #B5651D; + border-style: dotted; +} + /* Links */ a { color: var(--anchor); @@ -238,27 +260,32 @@ a.reference-toplevel { font-weight: bold; } +a.nimdoc { + word-spacing: 0.3em; +} + a.toc-backref { text-decoration: none; - color: var(--text); } + color: var(--text); +} a.link-seesrc { color: #607c9f; font-size: 0.9em; - font-style: italic; } + font-style: italic; +} -a:hover, -a:focus { +a:hover, a:focus { color: var(--anchor-focus); - text-decoration: underline; } + text-decoration: underline; +} a:hover span.Identifier { color: var(--anchor); } -sub, -sup { +sub, sup { position: relative; font-size: 75%; line-height: 0; @@ -285,8 +312,7 @@ img { background: transparent !important; box-shadow: none !important; } - a, - a:visited { + a, a:visited { text-decoration: underline; } a[href]:after { @@ -300,16 +326,14 @@ img { a[href^="#"]:after { content: ""; } - pre, - blockquote { + pre, blockquote { border: 1px solid #999; page-break-inside: avoid; } thead { display: table-header-group; } - tr, - img { + tr, img { page-break-inside: avoid; } img { @@ -324,22 +348,18 @@ img { h1.title { page-break-before: avoid; } - p, - h2, - h3 { + p, h2, h3 { orphans: 3; widows: 3; } - h2, - h3 { + h2, h3 { page-break-after: avoid; } } p { margin-top: 0.5em; - margin-bottom: 0.5em; -} + margin-bottom: 0.5em; } small { font-size: 85%; } @@ -347,8 +367,7 @@ small { strong { font-weight: 600; font-size: 0.95em; - color: var(--strong); -} + color: var(--strong); } em { font-style: italic; } @@ -369,14 +388,14 @@ h1.title { text-align: center; font-weight: 900; margin-top: 0.75em; - margin-bottom: 0em; -} + margin-bottom: 0em; } h2 { font-size: 1.3em; margin-top: 2em; } h2.subtitle { + margin-top: 0em; text-align: center; } h3 { @@ -396,36 +415,29 @@ h6 { font-size: 1.1em; } -ul, -ol { +ul, ol { padding: 0; margin-top: 0.5em; margin-left: 0.75em; } -ul ul, -ul ol, -ol ol, -ol ul { +ul ul, ul ol, ol ol, ol ul { margin-bottom: 0; margin-left: 1.25em; } -li { - list-style-type: circle; -} +ul.simple > li { + list-style-type: circle; } ul.simple-boot li { - list-style-type: none; - margin-left: 0em; - margin-bottom: 0.5em; -} + list-style-type: none; + margin-left: 0em; + margin-bottom: 0.5em; } ol.simple > li, ul.simple > li { - margin-bottom: 0.25em; + margin-bottom: 0.2em; margin-left: 0.4em } ul.simple.simple-toc > li { - margin-top: 1em; -} + margin-top: 1em; } ul.simple-toc { list-style: none; @@ -434,14 +446,21 @@ ul.simple-toc { margin-top: 1em; } ul.simple-toc > li { - list-style-type: none; -} + list-style-type: none; } ul.simple-toc-section { list-style-type: circle; - margin-left: 1em; + margin-left: 0.8em; color: #6c9aae; } +ul.nested-toc-section { + list-style-type: circle; + margin-left: -0.75em; + color: var(--text); } + +ul.nested-toc-section > li { + margin-left: 1.25em; } + ol.arabic { list-style: decimal; } @@ -480,6 +499,46 @@ hr { border: 0; border-top: 1px solid #aaa; } +hr.footnote { + width: 25%; + border-top: 0.15em solid #999; + margin-bottom: 0.15em; + margin-top: 0.15em; +} +div.footnote-group { + margin-left: 1em; +} +div.footnote-label { + display: inline-block; + min-width: 1.7em; +} + +div.option-list { + border: 0.1em solid var(--border); +} +div.option-list-item { + padding-left: 12em; + padding-right: 0; + padding-bottom: 0.3em; + padding-top: 0.3em; +} +div.odd { + background-color: var(--secondary-background); +} +div.option-list-label { + margin-left: -11.5em; + margin-right: 0em; + min-width: 11.5em; + display: inline-block; + vertical-align: top; +} +div.option-list-description { + width: calc(100% - 1em); + padding-left: 1em; + padding-right: 0; + display: inline-block; +} + blockquote { font-size: 0.9em; font-style: italic; @@ -488,7 +547,12 @@ blockquote { border-left: 5px solid #bbc; } -.pre { +blockquote.markdown-quote { + font-size: 0.9rem; /* use rem to avoid recursion */ + font-style: normal; +} + +.pre, span.tok { font-family: "Source Code Pro", Monaco, Menlo, Consolas, "Courier New", monospace; font-weight: 500; font-size: 0.85em; @@ -499,6 +563,16 @@ blockquote { border-radius: 4px; } +span.tok { + border: 1px solid #808080; + padding-bottom: 0.1em; + margin-right: 0.2em; +} + +.copyToClipBoard { + position: relative; +} + pre { font-family: "Source Code Pro", Monaco, Menlo, Consolas, "Courier New", monospace; color: var(--text); @@ -517,7 +591,25 @@ pre { border: 1px solid var(--border); -webkit-border-radius: 6px; -moz-border-radius: 6px; - border-radius: 6px; } + border-radius: 6px; +} + +.copyToClipBoardBtn { + visibility: hidden; + position: absolute; + width: 24px; + border-radius: 4px; + background-image: var(--clipboard-image); + right: 5px; + top: 13px; + background-color: var(--secondary-background); + padding: 11px; + border: 0; +} + +.copyToClipBoard:hover .copyToClipBoardBtn { + visibility: visible; +} .pre-scrollable { max-height: 340px; @@ -531,8 +623,8 @@ pre { table.line-nums-table { border-radius: 4px; - border: 1px solid #cccccc; - background-color: ghostwhite; + border: 1px solid var(--border); + background-color: var(--secondary-background); border-collapse: separate; margin-top: 15px; margin-bottom: 25px; } @@ -550,6 +642,7 @@ table.line-nums-table { .line-nums-table td.blob-line-nums pre { color: #b0b0b0; -webkit-filter: opacity(75%); + filter: opacity(75%); text-align: right; border-color: transparent; background-color: transparent; @@ -567,6 +660,9 @@ table { border-collapse: collapse; border-color: var(--third-background); border-spacing: 0; +} + +table:not(.line-nums-table) { font-size: 0.9em; } @@ -581,10 +677,11 @@ table th { font-weight: bold; } table th.docinfo-name { - background-color: transparent; + background-color: transparent; + text-align: right; } -table tr:hover { +table:not(.line-nums-table) tr:hover { background-color: var(--third-background); } @@ -597,6 +694,34 @@ table.borderless td, table.borderless th { The right padding separates the table cells. */ padding: 0 0.5em 0 0 !important; } +.admonition { + padding: 0.3em; + background-color: var(--secondary-background); + border-left: 0.4em solid #7f7f84; + margin-bottom: 0.5em; + -webkit-box-shadow: 0 5px 8px -6px rgba(0,0,0,.2); + -moz-box-shadow: 0 5px 8px -6px rgba(0,0,0,.2); + box-shadow: 0 5px 8px -6px rgba(0,0,0,.2); +} +.admonition-info { + border-color: var(--info-background); +} +.admonition-info-text { + color: var(--info-background); +} +.admonition-warning { + border-color: var(--warning-background); +} +.admonition-warning-text { + color: var(--warning-background); +} +.admonition-error { + border-color: var(--error-background); +} +.admonition-error-text { + color: var(--error-background); +} + .first { /* Override more specific margin styles with "! important". */ margin-top: 0 !important; } @@ -628,8 +753,7 @@ div.footer, div.header { font-size: smaller; } div.footer { - padding-top: 5em; -} + padding-top: 5em; } div.line-block { display: block; @@ -646,19 +770,23 @@ div.topic { div.search_results { background-color: var(--third-background); - margin: 3em; + margin: 3vh 5vw; padding: 1em; border: 1px solid #4d4d4d; -} + position: fixed; + top: 10px; + isolation: isolate; + max-width: calc(100vw - 6em); + z-index: 1; + max-height: calc(100vh - 6em); + overflow-y: scroll;} div#global-links ul { margin-left: 0; - list-style-type: none; -} + list-style-type: none; } div#global-links > simple-boot { - margin-left: 3em; -} + margin-left: 3em; } hr.docutils { width: 75%; } @@ -756,9 +884,6 @@ span.classifier { span.classifier-delimiter { font-weight: bold; } -span.option { - white-space: nowrap; } - span.problematic { color: #b30000; } @@ -838,8 +963,29 @@ span.Preprocessor { span.Directive { color: #252dbe; } -span.Command, span.Rule, span.Hyperlink, span.Label, span.Reference, -span.Other { +span.option { + font-weight: bold; + font-family: "Source Code Pro", Monaco, Menlo, Consolas, "Courier New", monospace; + color: var(--option); } + +span.Prompt { + font-weight: bold; + color: red; } + +span.ProgramOutput { + font-weight: bold; + color: #808080; } + +span.program { + font-weight: bold; + color: var(--program); + text-decoration: underline; + text-decoration-color: var(--hint); + text-decoration-thickness: 0.05em; + text-underline-offset: 0.15em; } + +span.Command, span.Rule, span.Hyperlink, +span.Label, span.Reference, span.Other { color: var(--other); } /* Pop type, const, proc, and iterator defs in nim def blocks */ @@ -860,6 +1006,7 @@ dt pre > span.Operator ~ span.Identifier, dt pre > span.Other ~ span.Identifier background-position: 0 0; background-size: 51px 14px; -webkit-filter: opacity(50%); + filter: opacity(50%); background-repeat: no-repeat; background-image: var(--nim-sprite-base64); margin-bottom: 5px; } @@ -876,17 +1023,14 @@ span.pragmadots { border-radius: 4px; margin: 0 2px; cursor: pointer; - font-size: 0.8em; -} + font-size: 0.8em; } span.pragmadots:hover { - background-color: var(--hint); -} + background-color: var(--hint); } + span.pragmawrap { - display: none; -} + display: none; } span.attachedType { display: none; - visibility: hidden; -} + visibility: hidden; } diff --git a/doc/nimfix.rst b/doc/nimfix.rst deleted file mode 100644 index 62064fe69..000000000 --- a/doc/nimfix.rst +++ /dev/null @@ -1,56 +0,0 @@ -===================== - Nimfix User Guide -===================== - -:Author: Andreas Rumpf -:Version: |nimversion| - -**WARNING**: Nimfix is currently beta-quality. - -Nimfix is a tool to help you upgrade from Nimrod (<= version 0.9.6) to -Nim (=> version 0.10.0). - -It performs 3 different actions: - -1. It makes your code case consistent. -2. It renames every symbol that has a deprecation rule. So if a module has a - rule ``{.deprecated: [TFoo: Foo].}`` then ``TFoo`` is replaced by ``Foo``. -3. It can also check that your identifiers adhere to the official style guide - and optionally modify them to do so (via ``--styleCheck:auto``). - -Note that ``nimfix`` defaults to **overwrite** your code unless you -use ``--overwriteFiles:off``! But hey, if you do not use a version control -system by this day and age, your project is already in big trouble. - - -Installation ------------- - -Nimfix is part of the compiler distribution. Compile via:: - - nim c compiler/nimfix/nimfix.nim - mv compiler/nimfix/nimfix bin - -Or on windows:: - - nim c compiler\nimfix\nimfix.nim - move compiler\nimfix\nimfix.exe bin - -Usage ------ - -Usage: - nimfix [options] projectfile.nim - -Options: - - --overwriteFiles:on|off overwrite the original nim files. DEFAULT is ON! - --wholeProject overwrite every processed file. - --checkExtern:on|off style check also extern names - --styleCheck:on|off|auto performs style checking for identifiers - and suggests an alternative spelling; - 'auto' corrects the spelling. - -In addition, all command line options of Nim are supported. - - diff --git a/doc/nimgrep.md b/doc/nimgrep.md new file mode 100644 index 000000000..63f760051 --- /dev/null +++ b/doc/nimgrep.md @@ -0,0 +1,128 @@ +========================= + nimgrep User's manual +========================= + +:Author: Andreas Rumpf +:Version: 1.6.0 + +.. default-role:: option +.. contents:: + +Nimgrep is a command line tool for search and replace tasks. It can search for +regex or peg patterns and can search whole directories at once. User +confirmation for every single replace operation can be requested. + +Nimgrep has particularly good support for Nim's +eccentric *style insensitivity* (see option `-y` below). +Apart from that it is a generic text manipulation tool. + + +Installation +============ + +Compile nimgrep with the command: + + ```cmd + nim c -d:release tools/nimgrep.nim + ``` + +And copy the executable somewhere in your ``$PATH``. + + +Command line switches +===================== + +.. include:: nimgrep_cmdline.txt + +Path filter options +------------------- + +Let us assume we have file `dirA/dirB/dirC/file.nim`. +Filesystem path options will match for these parts of the path: + +| option | matches for | +| :------------------ | :-------------------------------- | +| `--[not]extensions` | ``nim`` | +| `--[not]filename` | ``file.nim`` | +| `--[not]dirname` | ``dirA`` and ``dirB`` and ``dirC`` | +| `--[not]dirpath` | ``dirA/dirB/dirC`` | + +Combining multiple filter options together and negating them +------------------------------------------------------------ + +Options for filtering can be provided multiple times so they form a list, +which works as: +* positive filters + `--filename`, `--dirname`, `--dirpath`, `--inContext`, + `--inFile` accept files/matches if *any* pattern from the list is hit +* negative filters + `--notfilename`, `--notdirname`, `--notdirpath`, `--notinContext`, + `--notinFile` accept files/matches if *no* pattern from the list is hit. + +In other words the same filtering option repeated many times means logical OR. + +.. Important:: + Different filtering options are related by logical AND: they all must + be true for a match to be accepted. + E.g. `--filename:F --dirname:D1 --notdirname:D2` means + `filename(F) AND dirname(D1) AND (NOT dirname(D2))`. + +So negative filtering patterns are effectively related by logical OR also: +`(NOT PAT1) AND (NOT PAT2) == NOT (PAT1 OR PAT2)`:literal: in pseudo-code. + +That means you can always use only 1 such an option with logical OR, e.g. +`--notdirname:PAT1 --notdirname:PAT2` is fully equivalent to +`--notdirname:'PAT1|PAT2'`. + +.. Note:: + If you want logical AND on patterns you should compose 1 appropriate pattern, + possibly combined with multi-line mode `(?s)`:literal:. + E.g. to require that multi-line context of matches has occurrences of + **both** PAT1 and PAT2 use positive lookaheads (`(?=PAT)`:literal:): + ```cmd + nimgrep --inContext:'(?s)(?=.*PAT1)(?=.*PAT2)' + ``` + +Meaning of `^`:literal: and `$`:literal: +======================================== + +`nimgrep`:cmd: PCRE engine is run in a single-line mode so +`^`:literal: matches the beginning of whole input *file* and +`$`:literal: matches the end of *file* (or whole input *string* for +options like `--filename`). + +Add the `(?m)`:literal: modifier to the beginning of your pattern for +`^`:literal: and `$`:literal: to match the beginnings and ends of *lines*. + +Examples +======== + +All examples below use default PCRE Regex patterns: + ++ To search recursively in Nim files using style-insensitive identifiers: + + ```cmd + nimgrep --recursive --ext:'nim|nims' --ignoreStyle + # short: -r --ext:'nim|nims' -y + ``` + + .. Note:: we used `'` quotes to avoid special treatment of `|` symbol + for shells like Bash + ++ To exclude version control directories (Git, Mercurial=hg, Subversion=svn) + from the search: + ```cmd + nimgrep --notdirname:'^\.git$' --notdirname:'^\.hg$' --notdirname:'^\.svn$' + # short: --ndi:'^\.git$' --ndi:'^\.hg$' --ndi:'^\.svn$' + ``` ++ To search only in paths containing the `tests`:literal: sub-directory + recursively: + ```cmd + nimgrep --recursive --dirname:'^tests$' + # short: -r --di:'^tests$' + # or using --dirpath: + nimgrep --recursive --dirpath:'(^|/)tests($|/)' + # short: -r --pa:'(^|/)tests($|/)' + ``` ++ Nimgrep can search multi-line, e.g. to find files containing `import`:literal: + and then `strutils`:literal: use pattern `'import(.|\n)*?strutils'`:literal:. diff --git a/doc/nimgrep.rst b/doc/nimgrep.rst deleted file mode 100644 index 791ead162..000000000 --- a/doc/nimgrep.rst +++ /dev/null @@ -1,50 +0,0 @@ -========================= - nimgrep User's manual -========================= - -:Author: Andreas Rumpf -:Version: 0.9 - - -Nimgrep is a command line tool for search&replace tasks. It can search for -regex or peg patterns and can search whole directories at once. User -confirmation for every single replace operation can be requested. - -Nimgrep has particularly good support for Nim's -eccentric *style insensitivity*. Apart from that it is a generic text -manipulation tool. - - -Installation -============ - -Compile nimgrep with the command:: - - nim c -d:release tools/nimgrep.nim - -And copy the executable somewhere in your ``$PATH``. - - -Command line switches -===================== - -Usage: - nimgrep [options] [pattern] [replacement] (file/directory)* -Options: - --find, -f find the pattern (default) - --replace, -r replace the pattern - --peg pattern is a peg - --re pattern is a regular expression (default); extended - syntax for the regular expression is always turned on - --recursive process directories recursively - --confirm confirm each occurrence/replacement; there is a chance - to abort any time without touching the file - --stdin read pattern from stdin (to avoid the shell's confusing - quoting rules) - --word, -w the match should have word boundaries (buggy for pegs!) - --ignoreCase, -i be case insensitive - --ignoreStyle, -y be style insensitive - --ext:EX1|EX2|... only search the files with the given extension(s) - --verbose be verbose: list every processed file - --help, -h shows this help - --version, -v shows the version diff --git a/doc/nimgrep_cmdline.txt b/doc/nimgrep_cmdline.txt new file mode 100644 index 000000000..6f6887bc4 --- /dev/null +++ b/doc/nimgrep_cmdline.txt @@ -0,0 +1,136 @@ + +Usage: + +* To search: + + nimgrep [options] PATTERN [(FILE/DIRECTORY)*/-] + +* To replace: + + nimgrep [options] PATTERN --replace REPLACEMENT (FILE/DIRECTORY)*/- + +* To list file names: + + nimgrep [options] --filenames [PATTERN] [(FILE/DIRECTORY)*] + +Positional arguments, from left to right: +1) PATTERN is either Regex (default) or Peg if `--peg` is specified. + PATTERN and REPLACEMENT should be skipped when `--stdin` is specified. +2) REPLACEMENT supports `$1`, `$#` notations for captured groups in PATTERN. + + .. DANGER:: `--replace` mode **DOES NOT** ask confirmation + unless `--confirm` is specified! + +3) Final arguments are a list of paths (FILE/DIRECTORY) or a standalone + minus `-` or not specified (empty): + + * empty, current directory `.` is assumed (not with `--replace`) + + .. Note:: so when no FILE/DIRECTORY/`-` is specified nimgrep + does **not** read the pipe, but searches files in the current + dir instead! + * `-`, read buffer once from stdin: pipe or terminal input; + in `--replace` mode the result is directed to stdout; + it's not compatible with `--stdin`, `--filenames`, or `--confirm` + + + For any given DIRECTORY nimgrep searches only its immediate files without + traversing subdirectories unless `--recursive` is specified. + +In replacement mode we require all 3 positional arguments to avoid damaging. + +Options: +* Mode of operation: + --find, -f find the PATTERN (default) + --replace, -! replace the PATTERN to REPLACEMENT, rewriting the files + --confirm confirm each occurrence/replacement; there is a chance + to abort any time without touching the file + --filenames just list filenames. Provide a PATTERN to find it in + the filenames (not in the contents of a file) or run + with empty pattern to just list all files: + + nimgrep --filenames # In current dir + nimgrep --filenames "" DIRECTORY + # Note empty pattern "", lists all files in DIRECTORY +* Interprete patterns: + --peg PATTERN and PAT are Peg + --re PATTERN and PAT are regular expressions (default) + --rex, -x use the "extended" syntax for the regular expression + so that whitespace is not significant + --word, -w matches should have word boundaries (buggy for pegs!) + --ignoreCase, -i be case-insensitive in PATTERN and PAT + --ignoreStyle, -y be style insensitive in PATTERN and PAT + .. Note:: PATTERN and patterns PAT (see below in other options) are all either + Regex or Peg simultaneously and options `--rex`, `--word`, `--ignoreCase`, + and `--ignoreStyle` are applied to all of them. + +* File system walk: + --recursive, -r process directories recursively + --follow follow all symlinks when processing recursively + --sortTime, -s[:asc|desc] + order files by the last modification time (default: off): + ascending (recent files go last) or descending + +* Filter files (based on filesystem paths): + + .. Hint:: Instead of `not` you can type just `n` for negative options below. + + --ex[tensions]:EX1|EX2|... + only search the files with the given extension(s), + empty one (`--ex`) means files with missing extension + --notex[tensions]:EX1|EX2|... + exclude files having given extension(s), use empty one to + skip files with no extension (like some binary files are) + --fi[lename]:PAT search only files whose name matches pattern PAT + --notfi[lename]:PAT skip files whose name matches pattern PAT + --di[rname]:PAT select files that in their path have a directory name + that matches pattern PAT + --notdi[rname]:PAT do not descend into directories whose name (not path) + matches pattern PAT + --dirp[ath]:PAT select only files whose whole relative directory path + matches pattern PAT + --notdirp[ath]:PAT skip files whose whole relative directory path + matches pattern PAT + +* Filter files (based on file contents): + --inF[ile]:PAT select files containing a (not displayed) match of PAT + --notinF[ile]:PAT skip files containing a match of PAT + --bin:on|off|only process binary files? (detected by \0 in first 1K bytes) + (default: on - binary and text files treated the same way) + --text, -t process only text files, the same as `--bin:off` + +* Filter matches: + --inC[ontext]:PAT select only matches containing a match of PAT in their + surrounding context (multiline with `-c`, `-a`, `-b`) + --notinC[ontext]:PAT + skip matches not containing a match of PAT + in their surrounding context + +* Represent results: + --nocolor output will be given without any colors + --color[:on] force color even if output is redirected (default: auto) + --colorTheme:THEME select color THEME from `simple` (default), + `bnw` (black and white), `ack`, or `gnu` (GNU grep) + --count only print counts of matches for files that matched + --context:N, -c:N print N lines of leading context before every match and + N lines of trailing context after it (default N: 0) + --afterContext:N, -a:N + print N lines of trailing context after every match + --beforeContext:N, -b:N + print N lines of leading context before every match + --group, -g group matches by file + --newLine, -l display every matching line starting from a new line + --cols[:N] limit max displayed columns/width of output lines from + files by N characters, cropping overflows (default: off) + --cols:auto, -% calculate columns from terminal width for every line + --onlyAscii, -@ display only printable ASCII Latin characters 0x20-0x7E + substitutions: 0 -> ^@, 1 -> ^A, ... 0x1F -> ^_, + 0x7F -> '7F, ..., 0xFF -> 'FF + +* Miscellaneous: + --threads:N, -j:N speed up search by N additional workers (default: 0, off) + --stdin read PATTERN from stdin (to avoid the shell's confusing + quoting rules) and, if `--replace` given, REPLACEMENT + --verbose be verbose: list every processed file + --help, -h shows this help + --version, -v shows the version diff --git a/doc/niminst.rst b/doc/niminst.md index bf5cb0f50..cc399c57a 100644 --- a/doc/niminst.rst +++ b/doc/niminst.md @@ -5,6 +5,8 @@ :Author: Andreas Rumpf :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: Introduction @@ -12,7 +14,7 @@ Introduction niminst is a tool to generate an installer for a Nim program. Currently it can create an installer for Windows -via `Inno Setup <http://www.jrsoftware.org/isinfo.php>`_ as well as +via [Inno Setup](http://www.jrsoftware.org/isinfo.php) as well as installation/deinstallation scripts for UNIX. Later versions will support Linux' package management systems. @@ -24,15 +26,15 @@ systems. Configuration file ================== -niminst uses the Nim `parsecfg <parsecfg.html>`_ module to parse the +niminst uses the Nim [parsecfg](parsecfg.html) module to parse the configuration file. Here's an example of how the syntax looks like: .. include:: mytest.cfg :literal: The value of a key-value pair can reference user-defined variables via -the ``$variable`` notation: They can be defined in the command line with the -``--var:name=value`` switch. This is useful to not hard-coding the +the `$variable` notation: They can be defined in the command line with the +`--var:name=value`:option: switch. This is useful to not hard-coding the program's version number into the configuration file, for instance. It follows a description of each possible section and how it affects the @@ -47,96 +49,96 @@ contain the following key-value pairs: ==================== ======================================================= Key description ==================== ======================================================= -``Name`` the project's name; this needs to be a single word -``DisplayName`` the project's long name; this can contain spaces. If - not specified, this is the same as ``Name``. -``Version`` the project's version -``OS`` the OSes to generate C code for; for example: - ``"windows;linux;macosx"`` -``CPU`` the CPUs to generate C code for; for example: - ``"i386;amd64;powerpc"`` -``Authors`` the project's authors -``Description`` the project's description -``App`` the application's type: "Console" or "GUI". If +`Name` the project's name; this needs to be a single word +`DisplayName` the project's long name; this can contain spaces. If + not specified, this is the same as `Name`. +`Version` the project's version +`OS` the OSes to generate C code for; for example: + `"windows;linux;macosx"` +`CPU` the CPUs to generate C code for; for example: + `"i386;amd64;powerpc"` +`Authors` the project's authors +`Description` the project's description +`App` the application's type: "Console" or "GUI". If "Console", niminst generates a special batch file - for Windows to open up the command line shell. -``License`` the filename of the application's license + for Windows to open up the command-line shell. +`License` the filename of the application's license ==================== ======================================================= -``files`` key +`files` key ------------- -Many sections support the ``files`` key. Listed filenames -can be separated by semicolon or the ``files`` key can be repeated. Wildcards +Many sections support the `files` key. Listed filenames +can be separated by semicolon or the `files` key can be repeated. Wildcards in filenames are supported. If it is a directory name, all files in the -directory are used:: +directory are used: - [Config] - Files: "configDir" - Files: "otherconfig/*.conf;otherconfig/*.cfg" + [Config] + Files: "configDir" + Files: "otherconfig/*.conf;otherconfig/*.cfg" Config section -------------- -The ``config`` section currently only supports the ``files`` key. Listed files +The `config` section currently only supports the `files` key. Listed files will be installed into the OS's configuration directory. Documentation section --------------------- -The ``documentation`` section supports the ``files`` key. +The `documentation` section supports the `files` key. Listed files will be installed into the OS's native documentation directory (which might be ``$appdir/doc``). -There is a ``start`` key which determines whether the Windows installer +There is a `start` key which determines whether the Windows installer generates a link to e.g. the ``index.html`` of your documentation. Other section ------------- -The ``other`` section currently only supports the ``files`` key. +The `other` section currently only supports the `files` key. Listed files will be installed into the application installation directory -(``$appdir``). +(`$appdir`). Lib section ----------- -The ``lib`` section currently only supports the ``files`` key. +The `lib` section currently only supports the `files` key. Listed files will be installed into the OS's native library directory -(which might be ``$appdir/lib``). +(which might be `$appdir/lib`). Windows section --------------- -The ``windows`` section supports the ``files`` key for Windows specific files. +The `windows` section supports the `files` key for Windows-specific files. Listed files will be installed into the application installation directory -(``$appdir``). +(`$appdir`). Other possible options are: ==================== ======================================================= Key description ==================== ======================================================= -``BinPath`` paths to add to the Windows ``%PATH%`` environment +`BinPath` paths to add to the Windows `%PATH%` environment variable. Example: ``BinPath: r"bin;dist\mingw\bin"`` -``InnoSetup`` boolean flag whether an Inno Setup installer should be - generated for Windows. Example: ``InnoSetup: "Yes"`` +`InnoSetup` boolean flag whether an Inno Setup installer should be + generated for Windows. Example: `InnoSetup: "Yes"` ==================== ======================================================= UnixBin section --------------- -The ``UnixBin`` section currently only supports the ``files`` key. +The `UnixBin` section currently only supports the `files` key. Listed files will be installed into the OS's native bin directory (e.g. ``/usr/local/bin``). The exact location depends on the -installation path the user specifies when running the ``install.sh`` script. +installation path the user specifies when running the `install.sh` script. Unix section @@ -147,11 +149,11 @@ Possible options are: ==================== ======================================================= Key description ==================== ======================================================= -``InstallScript`` boolean flag whether an installation shell script - should be generated. Example: ``InstallScript: "Yes"`` -``UninstallScript`` boolean flag whether a deinstallation shell script +`InstallScript` boolean flag whether an installation shell script + should be generated. Example: `InstallScript: "Yes"` +`UninstallScript` boolean flag whether a de-installation shell script should be generated. - Example: ``UninstallScript: "Yes"`` + Example: `UninstallScript: "Yes"` ==================== ======================================================= @@ -163,10 +165,10 @@ Possible options are: ==================== ======================================================= Key description ==================== ======================================================= -``path`` Path to Inno Setup. +`path` Path to Inno Setup. Example: ``path = r"c:\inno setup 5\iscc.exe"`` -``flags`` Flags to pass to Inno Setup. - Example: ``flags = "/Q"`` +`flags` Flags to pass to Inno Setup. + Example: `flags = "/Q"` ==================== ======================================================= @@ -178,13 +180,13 @@ Possible options are: ==================== ======================================================= Key description ==================== ======================================================= -``path`` Path to the C compiler. -``flags`` Flags to pass to the C Compiler. - Example: ``flags = "-w"`` +`path` Path to the C compiler. +`flags` Flags to pass to the C Compiler. + Example: `flags = "-w"` ==================== ======================================================= -Real world example +Real-world example ================== The installers for the Nim compiler itself are generated by niminst. Have a diff --git a/doc/nims.rst b/doc/nims.md index 03afa258a..987cc2096 100644 --- a/doc/nims.rst +++ b/doc/nims.md @@ -2,31 +2,34 @@ NimScript ================================ -Strictly speaking, ``NimScript`` is the subset of Nim that can be evaluated +.. default-role:: code +.. include:: rstcommon.rst + +Strictly speaking, `NimScript` is the subset of Nim that can be evaluated by Nim's builtin virtual machine (VM). This VM is used for Nim's compiletime function evaluation features. -The ``nim`` executable processes the ``.nims`` configuration files in +The `nim`:cmd: executable processes the ``.nims`` configuration files in the following directories (in this order; later files overwrite previous settings): -1) If environment variable ``XDG_CONFIG_HOME`` is defined, +1) If environment variable `XDG_CONFIG_HOME` is defined, ``$XDG_CONFIG_HOME/nim/config.nims`` or ``~/.config/nim/config.nims`` (POSIX) or ``%APPDATA%/nim/config.nims`` (Windows). This file can be skipped - with the ``--skipUserCfg`` command line option. + with the `--skipUserCfg`:option: command line option. 2) ``$parentDir/config.nims`` where ``$parentDir`` stands for any parent directory of the project file's path. These files can be - skipped with the ``--skipParentCfg`` command line option. + skipped with the `--skipParentCfg`:option: command line option. 3) ``$projectDir/config.nims`` where ``$projectDir`` stands for the - project's path. This file can be skipped with the ``--skipProjCfg`` + project's path. This file can be skipped with the `--skipProjCfg`:option: command line option. 4) A project can also have a project specific configuration file named ``$project.nims`` that resides in the same directory as ``$project.nim``. This file can be skipped with the same - ``--skipProjCfg`` command line option. + `--skipProjCfg`:option: command line option. -For available procs and implementation details see `nimscript <nimscript.html>`_. +For available procs and implementation details see [nimscript](nimscript.html). Limitations @@ -36,13 +39,13 @@ NimScript is subject to some limitations caused by the implementation of the VM (virtual machine): * Nim's FFI (foreign function interface) is not available in NimScript. This - means that any stdlib module which relies on ``importc`` can not be used in + means that any stdlib module which relies on `importc` can not be used in the VM. -* ``ptr`` operations are are hard to emulate with the symbolic representation +* `ptr` operations are are hard to emulate with the symbolic representation the VM uses. They are available and tested extensively but there are bugs left. -* ``var T`` function arguments rely on ``ptr`` operations internally and might +* `var T` function arguments rely on `ptr` operations internally and might also be problematic in some cases. * More than one level of `ref` is generally not supported (for example, the type @@ -50,7 +53,7 @@ NimScript is subject to some limitations caused by the implementation of the VM * Multimethods are not available. -* ``random.randomize()`` requires an ``int64`` explicitly passed as argument, you *must* pass a Seed integer. +* `random.randomize()` requires an `int64` explicitly passed as argument, you *must* pass a Seed integer. Standard library modules @@ -58,96 +61,101 @@ Standard library modules At least the following standard library modules are available: -* `macros <macros.html>`_ -* `os <os.html>`_ -* `strutils <strutils.html>`_ -* `math <math.html>`_ -* `distros <distros.html>`_ -* `sugar <sugar.html>`_ -* `algorithm <algorithm.html>`_ -* `base64 <base64.html>`_ -* `bitops <bitops.html>`_ -* `chains <chains.html>`_ -* `colors <colors.html>`_ -* `complex <complex.html>`_ -* `htmlgen <htmlgen.html>`_ -* `httpcore <httpcore.html>`_ -* `lenientops <lenientops.html>`_ -* `mersenne <mersenne.html>`_ -* `options <options.html>`_ -* `parseutils <parseutils.html>`_ -* `punycode <punycode.html>`_ -* `random <punycode.html>`_ -* `stats <stats.html>`_ -* `strformat <strformat.html>`_ -* `strmisc <strmisc.html>`_ -* `strscans <strscans.html>`_ -* `unicode <unicode.html>`_ -* `uri <uri.html>`_ -* `std/editdistance <editdistance.html>`_ -* `std/wordwrap <wordwrap.html>`_ -* `std/sums <sums.html>`_ -* `parsecsv <parsecsv.html>`_ -* `parsecfg <parsecfg.html>`_ -* `parsesql <parsesql.html>`_ -* `xmlparser <xmlparser.html>`_ -* `htmlparser <htmlparser.html>`_ -* `ropes <ropes.html>`_ -* `json <json.html>`_ -* `parsejson <parsejson.html>`_ -* `strtabs <strtabs.html>`_ -* `unidecode <unidecode.html>`_ - -In addition to the standard Nim syntax (`system <system.html>`_ module), +* [algorithm](algorithm.html) +* [base64](base64.html) +* [bitops](bitops.html) +* [chains](chains.html) +* [colors](colors.html) +* [complex](complex.html) +* [distros](distros.html) +* [std/editdistance](editdistance.html) +* [htmlgen](htmlgen.html) +* [htmlparser](htmlparser.html) +* [httpcore](httpcore.html) +* [json](json.html) +* [lenientops](lenientops.html) +* [macros](macros.html) +* [math](math.html) +* [options](options.html) +* [os](os.html) +* [parsecfg](parsecfg.html) +* [parsecsv](parsecsv.html) +* [parsejson](parsejson.html) +* [parsesql](parsesql.html) +* [parseutils](parseutils.html) +* [punycode](punycode.html) +* [random](random.html) +* [ropes](ropes.html) +* [std/setutils](setutils.html) +* [stats](stats.html) +* [strformat](strformat.html) +* [strmisc](strmisc.html) +* [strscans](strscans.html) +* [strtabs](strtabs.html) +* [strutils](strutils.html) +* [sugar](sugar.html) +* [unicode](unicode.html) +* [unidecode](unidecode.html) +* [uri](uri.html) +* [std/wordwrap](wordwrap.html) +* [xmlparser](xmlparser.html) + +In addition to the standard Nim syntax ([system](system.html) module), NimScripts support the procs and templates defined in the -`nimscript <nimscript.html>`_ module too. +[nimscript](nimscript.html) module too. See also: -* `Check the tests for more information about modules compatible with NimScript. <https://github.com/nim-lang/Nim/blob/devel/tests/test_nimscript.nims>`_ +* [Check the tests for more information about modules compatible with NimScript]( + https://github.com/nim-lang/Nim/blob/devel/tests/test_nimscript.nims) NimScript as a configuration file ================================= -A command-line switch ``--FOO`` is written as ``switch("FOO")`` in -NimScript. Similarly, command-line ``--FOO:VAL`` translates to -``switch("FOO", "VAL")``. +A command-line switch `--FOO`:option: is written as `switch("FOO")` in +NimScript. Similarly, command-line `--FOO:VAL`:option: translates to +`switch("FOO", "VAL")`. -Here are few examples of using the ``switch`` proc: +Here are few examples of using the `switch` proc: -.. code-block:: nim + ```nim # command-line: --opt:size switch("opt", "size") - # command-line: --define:foo or -d:foo - switch("define", "foo") + # command-line: --define:release or -d:release + switch("define", "release") # command-line: --forceBuild switch("forceBuild") + # command-line: --hint[Conf]:off or --hint:Conf:off + switch("hint", "[Conf]:off") + ``` -NimScripts also support ``--`` templates for convenience, which look +NimScripts also support `--`:option: templates for convenience, which look like command-line switches written as-is in the NimScript file. So the above example can be rewritten as: -.. code-block:: nim + ```nim --opt:size - --define:foo + --define:release --forceBuild + ``` **Note**: In general, the *define* switches can also be set in -NimScripts using ``switch`` or ``--``, as shown in above -examples. Only the ``release`` define (``-d:release``) cannot be set -in NimScripts. +NimScripts using `switch` or `--`, as shown in above examples. Few +`define` switches such as `-d:strip`:option:, `-d:lto`:option: and +`-d:lto_incremental`:option: cannot be set in NimScripts. NimScript as a build tool ========================= -The ``task`` template that the ``system`` module defines allows a NimScript +The `task` template that the `system` module defines allows a NimScript file to be used as a build tool. The following example defines a -task ``build`` that is an alias for the ``c`` command: +task `build` that is an alias for the `c`:option: command: -.. code-block:: nim + ```nim task build, "builds an example": setCommand "c" + ``` In fact, as a convention the following tasks should be available: @@ -155,22 +163,22 @@ In fact, as a convention the following tasks should be available: ========= =================================================== Task Description ========= =================================================== -``help`` List all the available NimScript tasks along with their docstrings. -``build`` Build the project with the required - backend (``c``, ``cpp`` or ``js``). -``tests`` Runs the tests belonging to the project. -``bench`` Runs benchmarks belonging to the project. +`help` List all the available NimScript tasks along with their docstrings. +`build` Build the project with the required + backend (`c`:option:, `cpp`:option: or `js`:option:). +`tests` Runs the tests belonging to the project. +`bench` Runs benchmarks belonging to the project. ========= =================================================== -Look at the module `distros <distros.html>`_ for some support of the +Look at the module [distros](distros.html) for some support of the OS's native package managers. Nimble integration ================== -See the `Nimble readme <https://github.com/nim-lang/nimble#readme>`_ +See the [Nimble readme](https://github.com/nim-lang/nimble#readme) for more information. @@ -178,11 +186,10 @@ Standalone NimScript ==================== NimScript can also be used directly as a portable replacement for Bash and -Batch files. Use ``nim myscript.nims`` to run ``myscript.nims``. For example, +Batch files. Use `nim myscript.nims`:cmd: to run ``myscript.nims``. For example, installation of Nimble could be accomplished with this simple script: -.. code-block:: nim - + ```nim mode = ScriptMode.Verbose var id = 0 @@ -195,18 +202,19 @@ installation of Nimble could be accomplished with this simple script: exec "nim c nimble" mvFile "nimble" & $id & "/src/nimble".toExe, "bin/nimble".toExe + ``` -On Unix, you can also use the shebang ``#!/usr/bin/env nim``, as long as your filename +On Unix, you can also use the shebang `#!/usr/bin/env nim`, as long as your filename ends with ``.nims``: -.. code-block:: nim - + ```nim #!/usr/bin/env nim mode = ScriptMode.Silent echo "hello world" + ``` -Use ``#!/usr/bin/env -S nim --hints:off`` to disable hints. +Use `#!/usr/bin/env -S nim e --hints:off` to disable hints and relax the file extension constraint. Benefits @@ -226,9 +234,8 @@ allowing the same script to support a lot of systems. See the following (incomplete) example: -.. code-block:: nim - - import distros + ```nim + import std/distros # Architectures. if defined(amd64): @@ -253,6 +260,7 @@ See the following (incomplete) example: echo "Distro is ArchLinux" elif detectOs(Debian): echo "Distro is Debian" + ``` Uniform Syntax @@ -268,30 +276,30 @@ Powerful Metaprogramming NimScript can use Nim's templates, macros, types, concepts, effect tracking system, and more, you can create modules that work on compiled Nim and also on interpreted NimScript. -``func`` will still check for side effects, ``debugEcho`` also works as expected, +`func` will still check for side effects, `debugEcho` also works as expected, making it ideal for functional scripting metaprogramming. This is an example of a third party module that uses macros and templates to translate text strings on unmodified NimScript: -.. code-block:: nim - + ```nim import nimterlingua nimterlingua("translations.cfg") echo "cat" # Run with -d:RU becomes "kot", -d:ES becomes "gato", ... + ``` translations.cfg -.. code-block:: none - + ```none [cat] ES = gato - PT = minino + IT = gatto RU = kot FR = chat + ``` -* `Nimterlingua <https://nimble.directory/pkg/nimterlingua>`_ +* [Nimterlingua](https://nimble.directory/pkg/nimterlingua) Graceful Fallback @@ -302,8 +310,7 @@ but often a graceful and seamless fallback degradation is used. See the following NimScript: -.. code-block:: nim - + ```nim if likely(true): discard elif unlikely(false): @@ -313,9 +320,10 @@ See the following NimScript: static: echo CompileDate + ``` -``likely()``, ``unlikely()``, ``static:`` and ``{.compiletime.}`` +`likely()`, `unlikely()`, `static:` and `{.compiletime.}` will produce no code at all when run on NimScript, but still no error nor warning is produced and the code just works. @@ -323,14 +331,15 @@ Evolving Scripting language --------------------------- NimScript evolves together with Nim, -`occasionally new features might become available on NimScript <https://github.com/nim-lang/Nim/pulls?utf8=%E2%9C%93&q=nimscript>`_ , +[occasionally new features might become available on NimScript]( +https://github.com/nim-lang/Nim/pulls?q=nimscript+is%3Amerged), adapted from compiled Nim or added as new features on both. Scripting Language with a Package Manager ----------------------------------------- You can create your own modules to be compatible with NimScript, -and check `Nimble <https://nimble.directory>`_ +and check [Nimble](https://nimble.directory) to search for third party modules that may work on NimScript. DevOps Scripting @@ -339,4 +348,5 @@ DevOps Scripting You can use NimScript to deploy to production, run tests, build projects, do benchmarks, generate documentation, and all kinds of DevOps/SysAdmin specific tasks. -* `An example of a third party NimScript that can be used as a project-agnostic tool. <https://github.com/kaushalmodi/nim_config#list-available-tasks>`_ +* [An example of a third party NimScript that can be used as a project-agnostic + tool.](https://github.com/kaushalmodi/nim_config#list-available-tasks) diff --git a/doc/nimsuggest.rst b/doc/nimsuggest.md index a5d0d65b9..3d076a6f5 100644 --- a/doc/nimsuggest.rst +++ b/doc/nimsuggest.md @@ -5,61 +5,70 @@ :Author: Unknown :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: Nim differs from many other compilers in that it is really fast, and being so fast makes it suited to provide external queries for text editors about the source code being written. Through the -``nimsuggest`` tool, any IDE +`nimsuggest`:cmd: tool, any IDE can query a ``.nim`` source file and obtain useful information like definition of symbols or suggestions for completion. This document will guide you through the available options. If you want to look at practical examples of nimsuggest support you can look at the -`various editor integrations <https://github.com/Araq/Nim/wiki/Editor-Support>`_ +[various editor integrations](https://github.com/Araq/Nim/wiki/Editor-Support) already available. Installation ============ -Nimsuggest is part of Nim's core. Build it via:: +Nimsuggest is part of Nim's core. Build it via: + ```cmd koch nimsuggest + ``` Nimsuggest invocation ===================== -Run it via ``nimsuggest --stdin --debug myproject.nim``. Nimsuggest is a -server that takes queries that are related to ``myproject``. There is some +Run it via `nimsuggest --stdin --debug myproject.nim`:cmd:. Nimsuggest is a +server that takes queries that are related to `myproject`. There is some support so that you can throw random ``.nim`` files which are not part -of ``myproject`` at Nimsuggest too, but usually the query refer to modules/files -that are part of ``myproject``. +of `myproject` at Nimsuggest too, but usually the query refer to modules/files +that are part of `myproject`. -``--stdin`` means that Nimsuggest reads the query from ``stdin``. This is great +`--stdin`:option: means that Nimsuggest reads the query from `stdin`. This is great for testing things out and playing with it but for an editor communication via sockets is more reasonable so that is the default. It listens to port 6000 by default. +Nimsuggest is basically a frontend for the nim compiler so `--path`:option: flags and +[config files](nimc.html#compiler-usage-configuration-files) +can be used to specify additional dependencies like +`nimsuggest --stdin --debug --path:"dependencies" myproject.nim`:cmd:. + Specifying the location of the query ------------------------------------ Nimsuggest then waits for queries to process. A query consists of a -cryptic 3 letter "command" ``def`` or ``con`` or ``sug`` or ``use`` followed by +cryptic 3 letter "command" `def` or `con` or `sug` or `use` followed by a location. A query location consists of: ``file.nim`` - This is the name of the module or include file the query refers to. +: This is the name of the module or include file the query refers to. ``dirtyfile.nim`` - This is optional. +: This is optional. - The ``file`` parameter is enough for static analysis, but IDEs + The `file` parameter is enough for static analysis, but IDEs tend to have *unsaved buffers* where the user may still be in the middle of typing a line. In such situations the IDE can save the current contents to a temporary file and then use the @@ -68,18 +77,18 @@ a location. A query location consists of: ``line`` - An integer with the line you are going to query. For the compiler +: An integer with the line you are going to query. For the compiler lines start at **1**. ``col`` - An integer with the column you are going to query. For the +: An integer with the column you are going to query. For the compiler columns start at **0**. Definitions ----------- -The ``def`` Nimsuggest command performs a query about the definition +The `def` Nimsuggest command performs a query about the definition of a specific symbol. If available, Nimsuggest will answer with the type, source file, line/column information and other accessory data if available like a docstring. With this information an IDE can @@ -100,12 +109,12 @@ can't find any valid symbol matching the position of the query. Suggestions ----------- -The ``sug`` Nimsuggest command performs a query about possible +The `sug` Nimsuggest command performs a query about possible completion symbols at some point in the file. The typical usage scenario for this option is to call it after the -user has typed the dot character for `the object oriented call -syntax <tut2.html#object-oriented-programming-method-call-syntax>`_. +user has typed the dot character for [the object oriented call +syntax](tut2.html#object-oriented-programming-method-call-syntax). Nimsuggest will try to return the suggestions sorted first by scope (from innermost to outermost) and then by item name. @@ -113,7 +122,7 @@ Nimsuggest will try to return the suggestions sorted first by scope Invocation context ------------------ -The ``con`` Nimsuggest command is very similar to the suggestions +The `con` Nimsuggest command is very similar to the suggestions command, but instead of being used after the user has typed a dot character, this one is meant to be used after the user has typed an opening brace to start typing parameters. @@ -122,7 +131,7 @@ an opening brace to start typing parameters. Symbol usages ------------- -The ``use`` Nimsuggest command lists all usages of the symbol at +The `use` Nimsuggest command lists all usages of the symbol at a position. IDEs can use this to find all the places in the file where the symbol is used and offer the user to rename it in all places at the same time. @@ -140,15 +149,15 @@ Nimsuggest output is always returned on single lines separated by tab characters (``\t``). The values of each column are: 1. Three characters indicating the type of returned answer (e.g. - ``def`` for definition, ``sug`` for suggestion, etc). -2. Type of the symbol. This can be ``skProc``, ``skLet``, and just + `def` for definition, `sug` for suggestion, etc). +2. Type of the symbol. This can be `skProc`, `skLet`, and just about any of the enums defined in the module ``compiler/ast.nim``. 3. Fully qualified path of the symbol. If you are querying a symbol defined in the ``proj.nim`` file, this would have the form - ``proj.symbolName``. + `proj.symbolName`. 4. Type/signature. For variables and enums this will contain the type of the symbol, for procs, methods and templates this will - contain the full unique signature (e.g. ``proc (File)``). + contain the full unique signature (e.g. `proc (File)`). 5. Full path to the file containing the symbol. 6. Line where the symbol is located in the file. Lines start to count at **1**. diff --git a/doc/overview.rst b/doc/overview.md index e01520d7c..b21eb1e68 100644 --- a/doc/overview.rst +++ b/doc/overview.md @@ -5,5 +5,5 @@ Nim Documentation Overview :Author: Andreas Rumpf :Version: |nimversion| -.. include:: docs.rst +.. include:: docs.md diff --git a/doc/packaging.rst b/doc/packaging.md index ecde73f61..b742bef28 100644 --- a/doc/packaging.rst +++ b/doc/packaging.md @@ -4,7 +4,10 @@ Packaging Nim This page provide hints on distributing Nim using OS packages. -See `distros <distros.html>`_ for tools to detect Linux distribution at runtime. +See [distros](distros.html) for tools to detect Linux distribution at runtime. + +See [here](intern.html#bootstrapping-the-compiler-reproducible-builds) for how to +compile reproducible builds. Supported architectures ----------------------- @@ -45,28 +48,33 @@ The Debian package ships bash and ksh completion and manpages that can be reused Hints on the build process: -:: - - # build from C sources and then using koch - ./build.sh --os $os_type --cpu $cpu_arch - ./bin/nim c koch - ./koch boot -d:release + ```cmd + # build from C sources and then using koch + make -j # supports parallel build + # alternatively: ./build.sh --os $os_type --cpu $cpu_arch + ./bin/nim c -d:release koch + ./koch boot -d:release - # optionally generate docs into doc/html - ./koch docs + # optionally generate docs into doc/html + ./koch docs - ./koch tools -d:release + ./koch tools - # extract files to be really installed - ./install.sh <tempdir> + # extract files to be really installed + ./install.sh <tempdir> - # also include the tools - for fn in nimble nimsuggest nimgrep; do cp ./bin/$fn <tempdir>/nim/bin/; done + # also include the tools + for fn in nimble nimsuggest nimgrep; do cp ./bin/$fn <tempdir>/nim/bin/; done + ``` What to install: -- The expected stdlib location is /usr/lib/nim -- Global configuration files under /etc/nim +- The expected stdlib location is `/usr/lib/nim/lib`, previously it was just `/usr/lib/nim` +- `nimdoc.css` and `nimdoc.cls` from the `doc` folder should go into `/usr/lib/nim/doc/` +- `tools/debug/nim-gdb.py` should go into `/usr/lib/nim/tools/` +- `tools/dochack/dochack.js` should be installed to `/usr/lib/nim/tools/dochack/` +- Global configuration files under `/etc/nim` - Optionally: manpages, documentation, shell completion - When installing documentation, .idx files are not required - The "compiler" directory contains compiler sources and should not be part of the compiler binary package + diff --git a/doc/pegdocs.txt b/doc/pegdocs.txt index 4c557aed8..0a8fd8187 100644 --- a/doc/pegdocs.txt +++ b/doc/pegdocs.txt @@ -13,12 +13,12 @@ notation meaning ``A / ... / Z`` Ordered choice: Apply expressions `A`, ..., `Z`, in this order, to the text ahead, until one of them succeeds and possibly consumes some text. Indicate success if one of - expressions succeeded. Otherwise do not consume any text + expressions succeeded. Otherwise, do not consume any text and indicate failure. ``A ... Z`` Sequence: Apply expressions `A`, ..., `Z`, in this order, to consume consecutive portions of the text ahead, as long as they succeed. Indicate success if all succeeded. - Otherwise do not consume any text and indicate failure. + Otherwise, do not consume any text and indicate failure. The sequence's precedence is higher than that of ordered choice: ``A B / C`` means ``(A B) / Z`` and not ``A (B / Z)``. @@ -27,7 +27,10 @@ notation meaning ``{E}`` Capture: Apply expression `E` and store the substring that matched `E` into a *capture* that can be accessed after the matching process. -``$i`` Back reference to the ``i``th capture. ``i`` counts from 1. +``{}`` Empty capture: Delete the last capture. No character + is consumed. +``$i`` Back reference to the ``i``th capture. ``i`` counts forwards + from 1 or backwards (last capture to first) from ^1. ``$`` Anchor: Matches at the end of the input. No character is consumed. Same as ``!.``. ``^`` Anchor: Matches at the start of the input. No character @@ -41,20 +44,20 @@ notation meaning ``E+`` One or more: Apply expression `E` repeatedly to match the text ahead, as long as it succeeds. Consume the matched text (if any) and indicate success if there was at least - one match. Otherwise indicate failure. + one match. Otherwise, indicate failure. ``E*`` Zero or more: Apply expression `E` repeatedly to match the text ahead, as long as it succeeds. Consume the matched text (if any). Always indicate success. ``E?`` Zero or one: If expression `E` matches the text ahead, consume it. Always indicate success. ``[s]`` Character class: If the character ahead appears in the - string `s`, consume it and indicate success. Otherwise + string `s`, consume it and indicate success. Otherwise, indicate failure. ``[a-b]`` Character range: If the character ahead is one from the range `a` through `b`, consume it and indicate success. - Otherwise indicate failure. + Otherwise, indicate failure. ``'s'`` String: If the text ahead is the string `s`, consume it - and indicate success. Otherwise indicate failure. + and indicate success. Otherwise, indicate failure. ``i's'`` String match ignoring case. ``y's'`` String match ignoring style. ``v's'`` Verbatim string match: Use this to override a global @@ -63,15 +66,15 @@ notation meaning ``y$j`` String match ignoring style for back reference. ``v$j`` Verbatim string match for back reference. ``.`` Any character: If there is a character ahead, consume it - and indicate success. Otherwise (that is, at the end of + and indicate success. Otherwise, (that is, at the end of input) indicate failure. -``_`` Any Unicode character: If there is an UTF-8 character - ahead, consume it and indicate success. Otherwise indicate +``_`` Any Unicode character: If there is a UTF-8 character + ahead, consume it and indicate success. Otherwise, indicate failure. ``@E`` Search: Shorthand for ``(!E .)* E``. (Search loop for the pattern `E`.) ``{@} E`` Captured Search: Shorthand for ``{(!E .)*} E``. (Search - loop for the pattern `E`.) Everything until and exluding + loop for the pattern `E`.) Everything until and excluding `E` is captured. ``@@ E`` Same as ``{@} E``. ``A <- E`` Rule: Bind the expression `E` to the *nonterminal symbol* @@ -79,7 +82,7 @@ notation meaning matching engine.** ``\identifier`` Built-in macro for a longer expression. ``\ddd`` Character with decimal code *ddd*. -``\"``, etc Literal ``"``, etc. +``\"``, etc. Literal ``"``, etc. =============== ============================================================ @@ -128,51 +131,53 @@ notation meaning Supported PEG grammar --------------------- -The PEG parser implements this grammar (written in PEG syntax):: +The PEG parser implements this grammar (written in PEG syntax): - # Example grammar of PEG in PEG syntax. - # Comments start with '#'. - # First symbol is the start symbol. + # Example grammar of PEG in PEG syntax. + # Comments start with '#'. + # First symbol is the start symbol. - grammar <- rule* / expr + grammar <- rule* / expr - identifier <- [A-Za-z][A-Za-z0-9_]* - charsetchar <- "\\" . / [^\]] - charset <- "[" "^"? (charsetchar ("-" charsetchar)?)+ "]" - stringlit <- identifier? ("\"" ("\\" . / [^"])* "\"" / - "'" ("\\" . / [^'])* "'") - builtin <- "\\" identifier / [^\13\10] + identifier <- [A-Za-z][A-Za-z0-9_]* + charsetchar <- "\\" . / [^\]] + charset <- "[" "^"? (charsetchar ("-" charsetchar)?)+ "]" + stringlit <- identifier? ("\"" ("\\" . / [^"])* "\"" / + "'" ("\\" . / [^'])* "'") + builtin <- "\\" identifier / [^\13\10] - comment <- '#' @ \n - ig <- (\s / comment)* # things to ignore + comment <- '#' @ \n + ig <- (\s / comment)* # things to ignore - rule <- identifier \s* "<-" expr ig - identNoArrow <- identifier !(\s* "<-") - prefixOpr <- ig '&' / ig '!' / ig '@' / ig '{@}' / ig '@@' - literal <- ig identifier? '$' [0-9]+ / '$' / '^' / - ig identNoArrow / - ig charset / - ig stringlit / - ig builtin / - ig '.' / - ig '_' / - (ig "(" expr ig ")") - postfixOpr <- ig '?' / ig '*' / ig '+' - primary <- prefixOpr* (literal postfixOpr*) + rule <- identifier \s* "<-" expr ig + identNoArrow <- identifier !(\s* "<-") + prefixOpr <- ig '&' / ig '!' / ig '@' / ig '{@}' / ig '@@' + literal <- ig identifier? '$' '^'? [0-9]+ / '$' / '^' / + ig identNoArrow / + ig charset / + ig stringlit / + ig builtin / + ig '.' / + ig '_' / + (ig "(" expr ig ")") / + (ig "{" expr? ig "}") + postfixOpr <- ig '?' / ig '*' / ig '+' + primary <- prefixOpr* (literal postfixOpr*) - # Concatenation has higher priority than choice: - # ``a b / c`` means ``(a b) / c`` + # Concatenation has higher priority than choice: + # ``a b / c`` means ``(a b) / c`` - seqExpr <- primary+ - expr <- seqExpr (ig "/" expr)* + seqExpr <- primary+ + expr <- seqExpr (ig "/" expr)* **Note**: As a special syntactic extension if the whole PEG is only a single expression, identifiers are not interpreted as non-terminals, but are interpreted as verbatim string: -.. code-block:: nim + ```nim abc =~ peg"abc" # is true + ``` So it is not necessary to write ``peg" 'abc' "`` in the above example. @@ -182,22 +187,25 @@ Examples Check if `s` matches Nim's "while" keyword: -.. code-block:: nim + ```nim s =~ peg" y'while'" + ``` Exchange (key, val)-pairs: -.. code-block:: nim + ```nim "key: val; key2: val2".replacef(peg"{\ident} \s* ':' \s* {\ident}", "$2: $1") + ``` Determine the ``#include``'ed files of a C file: -.. code-block:: nim + ```nim for line in lines("myfile.c"): if line =~ peg"""s <- ws '#include' ws '"' {[^"]+} '"' ws comment <- '/*' @ '*/' / '//' .* ws <- (comment / \s+)* """: echo matches[0] + ``` PEG vs regular expression ------------------------- diff --git a/doc/readme.txt b/doc/readme.txt index 6f4cece87..1157752b9 100644 --- a/doc/readme.txt +++ b/doc/readme.txt @@ -1,7 +1,7 @@ ============================ -Nim's documenation system +Nim's documentation system ============================ This folder contains Nim's documentation. The documentation -is written in a format called *reStructuredText*, a markup language that reads +is written in a format called *Markdown*, a markup language that reads like ASCII and can be converted to HTML automatically! diff --git a/doc/refc.md b/doc/refc.md new file mode 100644 index 000000000..4023748e6 --- /dev/null +++ b/doc/refc.md @@ -0,0 +1,156 @@ +Tweaking the refc GC +==================== + +Cycle collector +--------------- + +The cycle collector can be en-/disabled independently from the other parts of +the garbage collector with `GC_enableMarkAndSweep` and `GC_disableMarkAndSweep`. + + +Soft real-time support +---------------------- + +To enable real-time support, the symbol `useRealtimeGC`:idx: needs to be +defined via `--define:useRealtimeGC`:option: (you can put this into your config +file as well). +With this switch the garbage collector supports the following operations: + + ```nim + proc GC_setMaxPause*(maxPauseInUs: int) + proc GC_step*(us: int, strongAdvice = false, stackSize = -1) + ``` + +The unit of the parameters `maxPauseInUs` and `us` is microseconds. + +These two procs are the two modus operandi of the real-time garbage collector: + +(1) GC_SetMaxPause Mode + + You can call `GC_SetMaxPause` at program startup and then each triggered + garbage collector run tries to not take longer than `maxPause` time. However, it is + possible (and common) that the work is nevertheless not evenly distributed + as each call to `new` can trigger the garbage collector and thus take `maxPause` + time. + +(2) GC_step Mode + + This allows the garbage collector to perform some work for up to `us` time. + This is useful to call in the main loop to ensure the garbage collector can do its work. + To bind all garbage collector activity to a `GC_step` call, + deactivate the garbage collector with `GC_disable` at program startup. + If `strongAdvice` is set to `true`, + then the garbage collector will be forced to perform the collection cycle. + Otherwise, the garbage collector may decide not to do anything, + if there is not much garbage to collect. + You may also specify the current stack size via `stackSize` parameter. + It can improve performance when you know that there are no unique Nim references + below a certain point on the stack. Make sure the size you specify is greater + than the potential worst-case size. + + It can improve performance when you know that there are no unique Nim + references below a certain point on the stack. Make sure the size you specify + is greater than the potential worst-case size. + +These procs provide a "best effort" real-time guarantee; in particular the +cycle collector is not aware of deadlines. Deactivate it to get more +predictable real-time behaviour. Tests show that a 1ms max pause +time will be met in almost all cases on modern CPUs (with the cycle collector +disabled). + + +Time measurement with garbage collectors +---------------------------------------- + +The garbage collectors' way of measuring time uses +(see ``lib/system/timers.nim`` for the implementation): + +1) `QueryPerformanceCounter` and `QueryPerformanceFrequency` on Windows. +2) `mach_absolute_time` on Mac OS X. +3) `gettimeofday` on Posix systems. + +As such it supports a resolution of nanoseconds internally; however, the API +uses microseconds for convenience. + +Define the symbol `reportMissedDeadlines` to make the +garbage collector output whenever it missed a deadline. +The reporting will be enhanced and supported by the API in later versions of the collector. + + +Tweaking the garbage collector +------------------------------ + +The collector checks whether there is still time left for its work after +every `workPackage`'th iteration. This is currently set to 100 which means +that up to 100 objects are traversed and freed before it checks again. Thus +`workPackage` affects the timing granularity and may need to be tweaked in +highly specialized environments or for older hardware. + + +Thread coordination +------------------- + +When the `NimMain()` function is called Nim initializes the garbage +collector to the current thread, which is usually the main thread of your +application. If your C code later spawns a different thread and calls Nim +code, the garbage collector will fail to work properly and you will crash. + +As long as you don't use the threadvar emulation Nim uses native thread +variables, of which you get a fresh version whenever you create a thread. You +can then attach a GC to this thread via + + ```nim + system.setupForeignThreadGc() + ``` + +It is **not** safe to disable the garbage collector and enable it after the +call from your background thread even if the code you are calling is short +lived. + +Before the thread exits, you should tear down the thread's GC to prevent memory +leaks by calling + + ```nim + system.tearDownForeignThreadGc() + ``` + + +Keeping track of memory +======================= + +If you need to pass around memory allocated by Nim to C, you can use the +procs `GC_ref` and `GC_unref` to mark objects as referenced to avoid them +being freed by the garbage collector. +Other useful procs from [system](system.html) you can use to keep track of memory are: + +* `getTotalMem()` Returns the amount of total memory managed by the garbage collector. +* `getOccupiedMem()` Bytes reserved by the garbage collector and used by objects. +* `getFreeMem()` Bytes reserved by the garbage collector and not in use. +* `GC_getStatistics()` Garbage collector statistics as a human-readable string. + +These numbers are usually only for the running thread, not for the whole heap, +with the exception of `--mm:boehm`:option: and `--mm:go`:option:. + +In addition to `GC_ref` and `GC_unref` you can avoid the garbage collector by manually +allocating memory with procs like `alloc`, `alloc0`, `allocShared`, `allocShared0` or `allocCStringArray`. +The garbage collector won't try to free them, you need to call their respective *dealloc* pairs +(`dealloc`, `deallocShared`, `deallocCStringArray`, etc) +when you are done with them or they will leak. + + + +Heap dump +========= + +The heap dump feature is still in its infancy, but it already proved +useful for us, so it might be useful for you. To get a heap dump, compile +with `-d:nimTypeNames`:option: and call `dumpNumberOfInstances` +at a strategic place in your program. +This produces a list of the used types in your program and for every type +the total amount of object instances for this type as well as the total +amount of bytes these instances take up. + +The numbers count the number of objects in all garbage collector heaps, they refer to +all running threads, not only to the current thread. (The current thread +would be the thread that calls `dumpNumberOfInstances`.) This might +change in later versions. diff --git a/doc/regexprs.txt b/doc/regexprs.txt index 83dbd2eeb..fa7f9d24a 100644 --- a/doc/regexprs.txt +++ b/doc/regexprs.txt @@ -47,16 +47,18 @@ full documentation of Perl's regular expressions. Because the backslash ``\`` is a meta character both in the Nim programming language and in regular expressions, it is strongly recommended that one uses the *raw* strings of Nim, so that -backslashes are interpreted by the regular expression engine:: +backslashes are interpreted by the regular expression engine: +```nim r"\S" # matches any character that is not whitespace +``` A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, -the pattern:: +the pattern: - The quick brown fox + The quick brown fox matches a portion of a subject string that is identical to itself. The power of regular expressions comes from the ability to include @@ -128,7 +130,7 @@ in patterns in a visible manner. There is no restriction on the appearance of non-printing characters, apart from the binary zero that terminates a pattern, but when a pattern is being prepared by text editing, it is usually easier to use one of the following escape sequences than the binary character it -represents:: +represents: ============== ============================================================ character meaning @@ -146,7 +148,7 @@ character meaning After ``\x``, from zero to two hexadecimal digits are read (letters can be in upper or lower case). In UTF-8 mode, any number of hexadecimal digits may appear between ``\x{`` and ``}``, but the value of the character code must be -less than 2**31 (that is, the maximum hexadecimal value is 7FFFFFFF). If +less than 2^31 (that is, the maximum hexadecimal value is 7FFFFFFF). If characters other than hexadecimal digits appear between ``\x{`` and ``}``, or if there is no terminating ``}``, this form of escape is not recognized. Instead, the initial ``\x`` will be interpreted as a basic hexadecimal escape, @@ -224,7 +226,7 @@ current matching point is at the end of the subject string, all of them fail, since there is no character to match. For compatibility with Perl, ``\s`` does not match the VT character (code 11). -This makes it different from the the POSIX "space" class. The ``\s`` characters +This makes it different from the POSIX "space" class. The ``\s`` characters are HT (9), LF (10), FF (12), CR (13), and space (32). A "word" character is an underscore or any character less than 256 that is @@ -244,7 +246,7 @@ The fourth use of backslash is for certain `simple assertions`:idx:. An assertion specifies a condition that has to be met at a particular point in a match, without consuming any characters from the subject string. The use of subpatterns for more complicated assertions is described below. The -backslashed assertions are:: +backslashed assertions are: ============== ============================================================ assertion meaning diff --git a/doc/rstcommon.rst b/doc/rstcommon.rst new file mode 100644 index 000000000..e9cc615db --- /dev/null +++ b/doc/rstcommon.rst @@ -0,0 +1,51 @@ +.. + Usage of this file: + Add this in the beginning of *.rst file: + + .. default-role:: code + .. include:: rstcommon.rst + + It's the current trick for brevity and compatibility with both Github and + rst2html.py, considering that Github cannot highlight Nim in + RST files anyway and it does not include files. + This way interpreted text is displayed with monospaced font in Github + and it's displayed an Nim code in both rst2html.py + (note ".. default-role:: Nim" above) and `nim rst2html`. + + For files that are user manual and consist of stuff like cmdline + option description, use 'code' as a **real** default role: + + .. include:: rstcommon.rst + .. default-role:: code + +.. define language roles explicitly (for compatibility with rst2html.py): + +.. role:: nim(code) + :language: nim + +.. default-role:: nim + +.. role:: c(code) + :language: c + +.. role:: cpp(code) + :language: cpp + +.. role:: yaml(code) + :language: yaml + +.. role:: python(code) + :language: python + +.. role:: java(code) + :language: java + +.. role:: csharp(code) + :language: csharp + +.. role:: cmd(code) + +.. role:: program(code) + +.. role:: option(code) + diff --git a/doc/sets_fragment.txt b/doc/sets_fragment.txt index 8436190a0..35b1fc023 100644 --- a/doc/sets_fragment.txt +++ b/doc/sets_fragment.txt @@ -1,27 +1,35 @@ The set type models the mathematical notion of a set. The set's basetype can only be an ordinal type of a certain size, namely: -* ``int8``-``int16`` -* ``uint8``/``byte``-``uint16`` -* ``char`` -* ``enum`` - -or equivalent. For signed integers the set's base type is defined to be in the -range ``0 .. MaxSetElements-1`` where ``MaxSetElements`` is currently always -2^16. +* `int8`-`int16` +* `uint8`/`byte`-`uint16` +* `char` +* `enum` +* Ordinal subrange types, i.e. `range[-10..10]` + +or equivalent. When constructing a set with signed integer literals, the set's +base type is defined to be in the range `0 .. DefaultSetElements-1` where +`DefaultSetElements` is currently always 2^8. The maximum range length for the +base type of a set is `MaxSetElements` which is currently always 2^16. Types +with a bigger range length are coerced into the range `0 .. MaxSetElements-1`. The reason is that sets are implemented as high performance bit vectors. Attempting to declare a set with a larger type will result in an error: -.. code-block:: nim +```nim + var s: set[int64] # Error: set is too large; use `std/sets` for ordinal types + # with more than 2^16 elements +``` + - var s: set[int64] # Error: set is too large +**Note:** Nim also offers [hash sets](sets.html) (which you need to import +with `import std/sets`), which have no such restrictions. -Sets can be constructed via the set constructor: ``{}`` is the empty set. The +Sets can be constructed via the set constructor: `{}` is the empty set. The empty set is type compatible with any concrete set type. The constructor can also be used to include elements (and ranges of elements): -.. code-block:: nim + ```nim type CharSet = set[char] var @@ -29,37 +37,44 @@ can also be used to include elements (and ranges of elements): x = {'a'..'z', '0'..'9'} # This constructs a set that contains the # letters from 'a' to 'z' and the digits # from '0' to '9' + ``` + +The module [`std/setutils`](setutils.html) provides a way to initialize a set from an iterable: + +```nim +import std/setutils + +let uniqueChars = myString.toSet +``` These operations are supported by sets: ================== ======================================================== operation meaning ================== ======================================================== -``A + B`` union of two sets -``A * B`` intersection of two sets -``A - B`` difference of two sets (A without B's elements) -``A == B`` set equality -``A <= B`` subset relation (A is subset of B or equal to B) -``A < B`` strict subset relation (A is a proper subset of B) -``e in A`` set membership (A contains element e) -``e notin A`` A does not contain element e -``contains(A, e)`` A contains element e -``card(A)`` the cardinality of A (number of elements in A) -``incl(A, elem)`` same as ``A = A + {elem}`` -``excl(A, elem)`` same as ``A = A - {elem}`` +`A + B` union of two sets +`A * B` intersection of two sets +`A - B` difference of two sets (A without B's elements) +`A == B` set equality +`A <= B` subset relation (A is subset of B or equal to B) +`A < B` strict subset relation (A is a proper subset of B) +`e in A` set membership (A contains element e) +`e notin A` A does not contain element e +`contains(A, e)` A contains element e +`card(A)` the cardinality of A (number of elements in A) +`incl(A, elem)` same as `A = A + {elem}` +`excl(A, elem)` same as `A = A - {elem}` ================== ======================================================== -Bit fields -~~~~~~~~~~ +### Bit fields Sets are often used to define a type for the *flags* of a procedure. This is a cleaner (and type safe) solution than defining integer -constants that have to be ``or``'ed together. +constants that have to be `or`'ed together. Enum, sets and casting can be used together as in: -.. code-block:: nim - + ```nim type MyFlag* {.size: sizeof(cint).} = enum A @@ -77,10 +92,11 @@ Enum, sets and casting can be used together as in: assert toNum({A, C}) == 5 assert toFlags(0) == {} assert toFlags(7) == {A, B, C} + ``` Note how the set turns enum values into powers of 2. If using enums and sets with C, use distinct cint. For interoperability with C see also the -`bitsize pragma <manual.html#implementation-specific-pragmas-bitsize-pragma>`_. +[bitsize pragma](manual.html#implementation-specific-pragmas-bitsize-pragma). diff --git a/doc/spawn.txt b/doc/spawn.txt index ab667ff48..ed25ad5fd 100644 --- a/doc/spawn.txt +++ b/doc/spawn.txt @@ -6,7 +6,7 @@ Nim has two flavors of parallelism: 1) `Structured`:idx parallelism via the ``parallel`` statement. 2) `Unstructured`:idx: parallelism via the standalone ``spawn`` statement. -Both need the `threadpool <threadpool.html>`_ module to work. +Both need the [threadpool](threadpool.html) module to work. Somewhat confusingly, ``spawn`` is also used in the ``parallel`` statement with slightly different semantics. ``spawn`` always takes a call expression of @@ -28,8 +28,8 @@ the passed expression on the thread pool and returns a `data flow variable`:idx: **blocking**. However, one can use ``blockUntilAny`` to wait on multiple flow variables at the same time: -.. code-block:: nim - import threadpool, ... + ```nim + import std/threadpool, ... # wait until 2 out of 3 servers received the update: proc main = @@ -40,6 +40,7 @@ variables at the same time: assert index >= 0 responses.del(index) discard blockUntilAny(responses) + ``` Data flow variables ensure that no data races are possible. Due to technical limitations not every type ``T`` is possible in @@ -54,9 +55,9 @@ Parallel statement Example: -.. code-block:: nim + ```nim # Compute PI in an inefficient way - import strutils, math, threadpool + import std/[strutils, math, threadpool] proc term(k: float): float = 4 * math.pow(-1, k) / (2*k + 1) @@ -69,6 +70,7 @@ Example: result += ch[k] echo formatFloat(pi(5000)) + ``` The parallel statement is the preferred mechanism to introduce parallelism diff --git a/doc/subexes.txt b/doc/subexes.txt index 54304f033..1bfd60213 100644 --- a/doc/subexes.txt +++ b/doc/subexes.txt @@ -14,7 +14,7 @@ Notation meaning ===================== ===================================================== ``$#`` use first or next argument ``$name`` use named argument, you can wrap the named argument - in curly braces (eg. ``${name}``) to separate it from + in curly braces (e.g. ``${name}``) to separate it from the next characters. ``$$`` produces a single ``$`` ``$1`` use first argument @@ -47,8 +47,7 @@ Notation meaning Examples ======== -.. code-block:: nim - + ```nim subex"$1($', '{2..})" % ["f", "a", "b", "c"] == "f(a, b, c)" subex"$1 $[files|file|files]{1} copied" % ["1"] == "1 file copied" @@ -57,5 +56,5 @@ Examples subex("type\n TEnum = enum\n $', '40c'\n '{..}") % [ "fieldNameA", "fieldNameB", "fieldNameC", "fieldNameD"] - + ``` diff --git a/doc/testament.md b/doc/testament.md new file mode 100644 index 000000000..0ff3591ac --- /dev/null +++ b/doc/testament.md @@ -0,0 +1,390 @@ +=========== + Testament +=========== + +.. default-role:: code +.. include:: rstcommon.rst +.. contents:: + +Testament is an advanced automatic unittests runner for Nim tests, is used for the development of Nim itself, +offers process isolation for your tests, it can generate statistics about test cases, +supports multiple targets (C, C++, ObjectiveC, JavaScript, etc.), +simulated [Dry-Runs](https://en.wikipedia.org/wiki/Dry_run_(testing)), +has logging, can generate HTML reports, skip tests from a file, and more, +so can be useful to run your tests, even the most complex ones. + + +Test files location +=================== + +By default, Testament looks for test files on ``"./tests/category/*.nim"``. +You can overwrite this pattern glob using `pattern <glob>`:option:. +The default working directory path can be changed using +`--directory:"folder/subfolder/"`:option:. + +Testament uses the `nim`:cmd: compiler on `PATH`. +You can change that using `--nim:"folder/subfolder/nim"`:option:. +Running JavaScript tests with `--targets:"js"`:option: requires +a working NodeJS on `PATH`. + +Commands +======== + +p|pat|pattern <glob> run all the tests matching the given pattern +all run all tests inside of category folders +c|cat|category <category> run all the tests of a certain category +r|run <test> run single test file +html generate testresults.html from the database + + +Options +======= + +--print print results to the console +--verbose print commands (compiling and running tests) +--simulate see what tests would be run but don't run them (for debugging) +--failing only show failing/ignored tests +--targets:"c cpp js objc" run tests for specified targets (default: c) +--nim:path use a particular nim executable (default: $PATH/nim) +--directory:dir Change to directory dir before reading the tests or doing anything else. +--colors:on|off Turn messages coloring on|off. +--backendLogging:on|off Disable or enable backend logging. By default turned on. +--megatest:on|off Enable or disable megatest. Default is on. +--valgrind:on|off Enable or disable valgrind support. Default is on. +--skipFrom:file Read tests to skip from `file` - one test per line, # comments ignored + + +Running a single test +===================== + +This is a minimal example to understand the basics, +not very useful for production, but easy to understand: + + ```console + $ mkdir -p tests/category + $ echo "assert 42 == 42" > tests/category/test0.nim + $ testament run tests/category/test0.nim + PASS: tests/category/test0.nim c ( 0.2 sec) + $ testament r tests/category/test0 + PASS: tests/category/test0.nim C ( 0.2 sec) + ``` + + +Running all tests from a directory +================================== + + This will run all tests in the top level tests/ directory. NOTE: these + tests are skipped by `testament all`. + + ```console + $ testament pattern "tests/*.nim" + ``` + +To search for tests deeper in a directory, use + + ```console + $ testament pattern "tests/**/*.nim" # one level deeper + $ testament pattern "tests/**/**/*.nim" # two levels deeper + ``` + +HTML Reports +============ + +Generate HTML Reports ``testresults.html`` from unittests, +you have to run at least 1 test *before* generating a report: + + ```console + $ testament html + ``` + + +Writing Unit tests +================== + +Example "template" **to edit** and write a Testament unittest: + + ```nim + discard """ + + # What actions to expect completion on. + # Options: + # "compile": expect successful compilation + # "run": expect successful compilation and execution + # "reject": expect failed compilation. The "reject" action can catch + # {.error.} pragmas but not {.fatal.} pragmas because + # {.error.} calls are expected to originate from the test-file, + # and can explicitly be specified using the "file", "line" and + # "column" options. + # {.fatal.} pragmas guarantee that compilation will be aborted. + action: "run" + + # For testing failed compilations you can specify the expected origin of the + # compilation error. + # With the "file", "line" and "column" options you can define the file, + # line and column that a compilation-error should have originated from. + # Use only with action: "reject" as it expects a failed compilation. + # Requires errormsg or msg to be defined. + # file: "" + # line: "" + # column: "" + + # The exit code that the test is expected to return. Typically, the default + # value of 0 is fine. Note that if the test will be run by valgrind, then + # the test will exit with either a code of 0 on success or 1 on failure. + exitcode: 0 + + # Provide an `output` string to assert that the test prints to standard out + # exactly the expected string. Provide an `outputsub` string to assert that + # the string given here is a substring of the standard out output of the + # test (the output includes both the compiler and test execution output). + output: "" + outputsub: "" + + # Whether to sort the compiler output lines before comparing them to the + # expected output. + sortoutput: true + + # Provide a `nimout` string to assert that the compiler during compilation + # prints the defined lines to the standard out. + # The lines must match in order, but there may be more lines that appear + # before, after, or in between them. + nimout: ''' + a very long, + multi-line + string''' + + # This is the Standard Input the test should take, if any. + input: "" + + # Error message the test should print, if any. + errormsg: "" + + # Can be run in batch mode, or not. + batchable: true + + # Can be run Joined with other tests to run all together, or not. + joinable: true + + # On Linux 64-bit machines, whether to use Valgrind to check for bad memory + # accesses or memory leaks. On other architectures, the test will be run + # as-is, without Valgrind. + # Options: + # true: run the test with Valgrind + # false: run the without Valgrind + # "leaks": run the test with Valgrind, but do not check for memory leaks + valgrind: false # Can use Valgrind to check for memory leaks, or not (Linux 64Bit only). + + # Checks that the specified piece of C-code is within the generated C-code. + ccodecheck: "'Assert error message'" + + # Command the test should use to run. If left out or an empty string is + # provided, the command is taken to be: + # "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file" + # Subject to variable interpolation. + cmd: "nim c -r $file" + + # Maximum generated temporary intermediate code file size for the test. + maxcodesize: 666 + + # Timeout seconds to run the test. Fractional values are supported. + timeout: 1.5 + + # Targets to run the test into (c, cpp, objc, js). Defaults to c. + targets: "c js" + + # flags with which to run the test, delimited by `;` + matrix: "; -d:release; -d:caseFoo -d:release" + + # Conditions that will skip this test. Use of multiple "disabled" clauses + # is permitted. + disabled: "bsd" # Can disable OSes... + disabled: "win" + disabled: "32bit" # ...or architectures + disabled: "i386" + disabled: "azure" # ...or pipeline runners + disabled: true # ...or can disable the test entirely + + """ + assert true + assert 42 == 42, "Assert error message" + ``` + + +* As you can see the "Spec" is just a `discard """ """`. +* Spec has sane defaults, so you don't need to provide them all, any simple assert will work just fine. +* This is not the full spec of Testament, check [the Testament Spec on GitHub, + see parseSpec()](https://github.com/nim-lang/Nim/blob/devel/testament/specs.nim#L317). +* Nim itself uses Testament, so [there are plenty of test examples]( + https://github.com/nim-lang/Nim/tree/devel/tests). +* Has some built-in CI compatibility, like Azure Pipelines, etc. + + +Inline hints, warnings and errors (notes) +----------------------------------------- + +Testing the line, column, kind and message of hints, warnings and errors can +be written inline like so: + ```nim + {.warning: "warning!!"} #[tt.Warning + ^ warning!! [User] ]# + ``` + +The opening `#[tt.` marks the message line. +The `^` marks the message column. + +Inline messages can be combined with `nimout` when `nimoutFull` is false (default). +This allows testing for expected messages from other modules: + + ```nim + discard """ + nimout: "config.nims(1, 1) Hint: some hint message [User]" + """ + {.warning: "warning!!"} #[tt.Warning + ^ warning!! [User] ]# + ``` + +Multiple messages for a line can be checked by delimiting messages with ';': + + ```nim + discard """ + matrix: "--errorMax:0 --styleCheck:error" + """ + + proc generic_proc*[T](a_a: int) = #[tt.Error + ^ 'generic_proc' should be: 'genericProc'; tt.Error + ^ 'a_a' should be: 'aA' ]# + discard + ``` + +Use `--errorMax:0` in `matrix`, or `cmd: "nim check $file"` when testing +for multiple 'Error' messages. + +Output message variable interpolation +------------------------------------- + +`errormsg`, `nimout`, and inline messages are subject to these variable interpolations: + +* `${/}` - platform's directory separator +* `$file` - the filename (without directory) of the test + +All other `$` characters need escaped as `$$`. + +Cmd variable interpolation +-------------------------- + +The `cmd` option is subject to these variable interpolations: + +* `$target` - the compilation target, e.g. `c`. +* `$options` - the options for the compiler. +* `$file` - the file path of the test. +* `$filedir` - the directory of the test file. + +.. code-block:: nim + + discard """ + cmd: "nim $target --nimblePath:./nimbleDir/simplePkgs $options $file" + """ + +All other `$` characters need escaped as `$$`. + + +Unit test Examples +================== + +Expected to fail: + + ```nim + discard """ + errormsg: "undeclared identifier: 'not_defined'" + """ + assert not_defined == "not_defined", "not_defined is not defined" + ``` + +Expected to fail with error thrown from another file: +```nim +# test.nim +discard """ + action: "reject" + errorMsg: "I break" + file: "breakPragma.nim" +""" +import ./breakPragma + +proc x() {.justDo.} = discard + +# breakPragma.nim +import std/macros + +macro justDo*(procDef: typed): untyped = + error("I break") + return procDef +``` + +Expecting generated C to contain a given piece of code: + + ```nim + discard """ + # Checks that the string "Assert error message" is in the generated + # C code. + ccodecheck: "'Assert error message'" + """ + assert 42 == 42, "Assert error message" + ``` + + +Non-Zero exit code: + + ```nim + discard """ + exitcode: 1 + """ + quit "Non-Zero exit code", 1 + ``` + +Standard output checking: + + ```nim + discard """ + + output: ''' + 0 + 1 + 2 + 3 + 4 + 5 + ''' + + """ + for i in 0..5: echo i + ``` + +JavaScript tests: + + ```nim + discard """ + targets: "js" + """ + when defined(js): + import std/jsconsole + console.log("My Frontend Project") + ``` + +Compile-time tests: + + ```nim + discard """ + action: "compile" + """ + static: assert 9 == 9, "Compile time assert" + ``` + +Tests without Spec: + + ```nim + assert 1 == 1 + ``` + + +See also: +* [Unittest](unittest.html) diff --git a/doc/tools.md b/doc/tools.md new file mode 100644 index 000000000..baf7ce386 --- /dev/null +++ b/doc/tools.md @@ -0,0 +1,45 @@ +======================== +Tools available with Nim +======================== + +.. default-role:: code +.. include:: rstcommon.rst + +The standard distribution ships with the following tools: + +- | [atlas](atlas.html) + | `atlas`:cmd: is a simple package cloner tool. It manages an isolated workspace that + contains projects and dependencies. + +- | [Hot code reloading](hcr.html) + | The "Hot code reloading" feature is built into the compiler but has its own + document explaining how it works. + +- | [Documentation generator](docgen.html) + | The builtin document generator `nim doc`:cmd: generates HTML documentation + from ``.nim`` source files. + +- | [Nimsuggest for IDE support](nimsuggest.html) + | Through the `nimsuggest`:cmd: tool, any IDE can query a ``.nim`` source file + and obtain useful information like the definition of symbols or suggestions for + completion. + +- | [C2nim](https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst) + | C to Nim source converter. Translates C header files to Nim. + +- | [niminst](niminst.html) + | niminst is a tool to generate an installer for a Nim program. + +- | [nimgrep](nimgrep.html) + | Nim search and replace utility. + +- | nimpretty + | `nimpretty`:cmd: is a Nim source code beautifier, + to format code according to the official style guide. + +- | [testament](testament.html) + | `testament`:cmd: is an advanced automatic *unittests runner* for Nim tests, + is used for the development of Nim itself, offers process isolation for your tests, + it can generate statistics about test cases, supports multiple targets (C, JS, etc), + has logging, can generate HTML reports, skip tests from a file, and more, + so can be useful to run your tests, even the most complex ones. diff --git a/doc/tools.rst b/doc/tools.rst deleted file mode 100644 index 66cbdca26..000000000 --- a/doc/tools.rst +++ /dev/null @@ -1,39 +0,0 @@ -======================== -Tools available with Nim -======================== - -The standard distribution ships with the following tools: - -- | `Hot code reloading <hcr.html>`_ - | The "Hot code reloading" feature is built into the compiler but has its own - document explaining how it works. - -- | `Documentation generator <docgen.html>`_ - | The builtin document generator ``nim doc`` generates HTML documentation - from ``.nim`` source files. - -- | `Nimsuggest for IDE support <nimsuggest.html>`_ - | Through the ``nimsuggest`` tool, any IDE can query a ``.nim`` source file - and obtain useful information like definition of symbols or suggestions for - completion. - -- | `C2nim <https://github.com/nim-lang/c2nim/blob/master/doc/c2nim.rst>`_ - | C to Nim source converter. Translates C header files to Nim. - -- | `niminst <niminst.html>`_ - | niminst is a tool to generate an installer for a Nim program. - -- | `nimgrep <nimgrep.html>`_ - | Nim search and replace utility. - -- | nimpretty - | ``nimpretty`` is a Nim source code beautifier, - to format code according to the official style guide. - -- | testament - | ``testament`` is an advanced automatic *unittests runner* for Nim tests, - is used for the development of Nim itself, offers process isolation for your tests, - it can generate statistics about test cases, supports multiple targets (C, JS, etc), - `simulated Dry-Runs <https://en.wikipedia.org/wiki/Dry_run_(testing)>`_, - has logging, can generate HTML reports, skip tests from a file and more, - so can be useful to run your tests, even the most complex ones. diff --git a/doc/tut1.rst b/doc/tut1.md index 24874096c..2e83effa3 100644 --- a/doc/tut1.rst +++ b/doc/tut1.md @@ -5,23 +5,31 @@ Nim Tutorial (Part I) :Author: Andreas Rumpf :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: Introduction ============ -.. raw:: html - <blockquote><p> - "Der Mensch ist doch ein Augentier -- schöne Dinge wünsch ich mir." - </p></blockquote> +> "Der Mensch ist doch ein Augentier -- Schöne Dinge wünsch' ich mir." This document is a tutorial for the programming language *Nim*. + This tutorial assumes that you are familiar with basic programming concepts -like variables, types or statements but is kept very basic. The `manual -<manual.html>`_ contains many more examples of the advanced language features. +like variables, types, or statements. + +Here are several other resources for learning Nim: + +* [Nim Basics tutorial](https://narimiran.github.io/nim-basics/) - a gentle + introduction of the concepts mentioned above +* [Learn Nim in 5 minutes](https://learnxinyminutes.com/docs/nim/) - quick, + five-minute introduction to Nim +* [The Nim manual](manual.html) - many more examples of the advanced language features + All code examples in this tutorial, as well as the ones found in the rest of -Nim's documentation, follow the `Nim style guide <nep1.html>`_. +Nim's documentation, follow the [Nim style guide](nep1.html). The first program @@ -29,59 +37,69 @@ The first program We start the tour with a modified "hello world" program: -.. code-block:: Nim - :test: "nim c $1" + ```Nim test = "nim c $1" # This is a comment echo "What's your name? " var name: string = readLine(stdin) echo "Hi, ", name, "!" + ``` -Save this code to the file "greetings.nim". Now compile and run it:: - +Save this code to the file "greetings.nim". Now compile and run it: + ```cmd nim compile --run greetings.nim + ``` -With the ``--run`` `switch <nimc.html#compiler-usage-command-line-switches>`_ Nim +With the ``--run`` [switch](nimc.html#compiler-usage-commandminusline-switches) Nim executes the file automatically after compilation. You can give your program -command line arguments by appending them after the filename:: - +command-line arguments by appending them after the filename: + ```cmd nim compile --run greetings.nim arg1 arg2 + ``` -Commonly used commands and switches have abbreviations, so you can also use:: - +Commonly used commands and switches have abbreviations, so you can also use: + ```cmd nim c -r greetings.nim + ``` -To compile a release version use:: - +This is a **debug version**. +To compile a release version use: + ```cmd nim c -d:release greetings.nim + ``` -By default the Nim compiler generates a large amount of runtime checks +By default, the Nim compiler generates a large number of runtime checks aiming for your debugging pleasure. With ``-d:release`` some checks are -`turned off and optimizations are turned on -<nimc.html#compiler-usage-compile-time-symbols>`_. +[turned off and optimizations are turned on]( +nimc.html#compiler-usage-compileminustime-symbols). + +For benchmarking or production code, use the ``-d:release`` switch. +For comparing the performance with unsafe languages like C, use the ``-d:danger`` switch +in order to get meaningful, comparable results. Otherwise, Nim might be handicapped +by checks that are **not even available** for C. Though it should be pretty obvious what the program does, I will explain the syntax: statements which are not indented are executed when the program starts. Indentation is Nim's way of grouping statements. Indentation is done with spaces only, tabulators are not allowed. -String literals are enclosed in double quotes. The ``var`` statement declares -a new variable named ``name`` of type ``string`` with the value that is -returned by the `readLine <io.html#readLine,File>`_ procedure. Since the -compiler knows that `readLine <io.html#readLine,File>`_ returns a string, +String literals are enclosed in double-quotes. The `var` statement declares +a new variable named `name` of type `string` with the value that is +returned by the [readLine](syncio.html#readLine,File) procedure. Since the +compiler knows that [readLine](syncio.html#readLine,File) returns a string, you can leave out the type in the declaration (this is called `local type inference`:idx:). So this will work too: -.. code-block:: Nim - :test: "nim c $1" + ```Nim test = "nim c $1" var name = readLine(stdin) + ``` Note that this is basically the only form of type inference that exists in Nim: it is a good compromise between brevity and readability. The "hello world" program contains several identifiers that are already known -to the compiler: ``echo``, `readLine <io.html#readLine,File>`_, etc. -These built-ins are declared in the system_ module which is implicitly +to the compiler: `echo`, [readLine](syncio.html#readLine,File), etc. +These built-ins are declared in the [system](system.html) module which is implicitly imported by any other module. @@ -96,17 +114,18 @@ keywords, comments, operators, and other punctuation marks. String and character literals ----------------------------- -String literals are enclosed in double quotes; character literals in single +String literals are enclosed in double-quotes; character literals in single quotes. Special characters are escaped with ``\``: ``\n`` means newline, ``\t`` means tabulator, etc. There are also *raw* string literals: -.. code-block:: Nim + ```Nim r"C:\program files\nim" + ``` -In raw literals the backslash is not an escape character. +In raw literals, the backslash is not an escape character. -The third and last way to write string literals are *long string literals*. -They are written with three quotes: ``""" ... """``; they can span over +The third and last way to write string literals is *long-string literals*. +They are written with three quotes: `""" ... """`; they can span over multiple lines and the ``\`` is not an escape character either. They are very useful for embedding HTML code templates for example. @@ -115,24 +134,23 @@ Comments -------- Comments start anywhere outside a string or character literal with the -hash character ``#``. Documentation comments start with ``##``: +hash character `#`. Documentation comments start with `##`: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" # A comment. var myVariable: int ## a documentation comment + ``` Documentation comments are tokens; they are only allowed at certain places in the input file as they belong to the syntax tree! This feature enables simpler documentation generators. -Multiline comments are started with ``#[`` and terminated with ``]#``. Multiline +Multiline comments are started with `#[` and terminated with `]#`. Multiline comments can also be nested. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" #[ You can have any Nim code text commented out inside this with no indentation restrictions. @@ -141,16 +159,17 @@ comments can also be nested. Note: these can be nested!! ]# ]# + ``` Numbers ------- Numerical literals are written as in most other languages. As a special twist, -underscores are allowed for better readability: ``1_000_000`` (one million). -A number that contains a dot (or 'e' or 'E') is a floating point literal: -``1.0e9`` (one billion). Hexadecimal literals are prefixed with ``0x``, -binary literals with ``0b`` and octal literals with ``0o``. A leading zero +underscores are allowed for better readability: `1_000_000` (one million). +A number that contains a dot (or 'e' or 'E') is a floating-point literal: +`1.0e9` (one billion). Hexadecimal literals are prefixed with `0x`, +binary literals with `0b` and octal literals with `0o`. A leading zero alone does not produce an octal. @@ -158,49 +177,19 @@ The var statement ================= The var statement declares a new local or global variable: -.. code-block:: - var x, y: int # declares x and y to have the type ``int`` + ```nim + var x, y: int # declares x and y to have the type `int` + ``` -Indentation can be used after the ``var`` keyword to list a whole section of +Indentation can be used after the `var` keyword to list a whole section of variables: -.. code-block:: - :test: "nim c $1" + ```nim test = "nim c $1" var x, y: int # a comment can occur here too a, b, c: string - - -The assignment statement -======================== - -The assignment statement assigns a new value to a variable or more generally -to a storage location: - -.. code-block:: - var x = "abc" # introduces a new variable `x` and assigns a value to it - x = "xyz" # assigns a new value to `x` - -``=`` is the *assignment operator*. The assignment operator can be -overloaded. You can declare multiple variables with a single assignment -statement and all the variables will have the same value: - -.. code-block:: - :test: "nim c $1" - var x, y = 3 # assigns 3 to the variables `x` and `y` - echo "x ", x # outputs "x 3" - echo "y ", y # outputs "y 3" - x = 42 # changes `x` to 42 without changing `y` - echo "x ", x # outputs "x 42" - echo "y ", y # outputs "y 3" - -Note that declaring multiple variables with a single assignment which calls a -procedure can have unexpected results: the compiler will *unroll* the -assignments and end up calling the procedure several times. If the result of -the procedure depends on side effects, your variables may end up having -different values! For safety use side-effect free procedures if making multiple -assignments. + ``` Constants @@ -210,42 +199,69 @@ Constants are symbols which are bound to a value. The constant's value cannot change. The compiler must be able to evaluate the expression in a constant declaration at compile time: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" const x = "abc" # the constant x contains the string "abc" + ``` -Indentation can be used after the ``const`` keyword to list a whole section of +Indentation can be used after the `const` keyword to list a whole section of constants: -.. code-block:: - :test: "nim c $1" + ```nim test = "nim c $1" const x = 1 # a comment can occur here too y = 2 z = y + 5 # computations are possible + ``` The let statement ================= -The ``let`` statement works like the ``var`` statement but the declared +The `let` statement works like the `var` statement but the declared symbols are *single assignment* variables: After the initialization their value cannot change: -.. code-block:: + ```nim let x = "abc" # introduces a new variable `x` and binds a value to it x = "xyz" # Illegal: assignment to `x` + ``` -The difference between ``let`` and ``const`` is: ``let`` introduces a variable -that can not be re-assigned, ``const`` means "enforce compile time evaluation +The difference between `let` and `const` is: `let` introduces a variable +that can not be re-assigned, `const` means "enforce compile time evaluation and put it into a data section": -.. code-block:: + ```nim const input = readLine(stdin) # Error: constant expression expected + ``` -.. code-block:: - :test: "nim c $1" + ```nim test = "nim c $1" let input = readLine(stdin) # works + ``` + + +The assignment statement +======================== + +The assignment statement assigns a new value to a variable or more generally +to a storage location: + + ```nim + var x = "abc" # introduces a new variable `x` and assigns a value to it + x = "xyz" # assigns a new value to `x` + ``` + +`=` is the *assignment operator*. The assignment operator can be +overloaded. You can declare multiple variables with a single assignment +statement and all the variables will have the same value: + + ```nim test = "nim c $1" + var x, y = 3 # assigns 3 to the variables `x` and `y` + echo "x ", x # outputs "x 3" + echo "y ", y # outputs "y 3" + x = 42 # changes `x` to 42 without changing `y` + echo "x ", x # outputs "x 42" + echo "y ", y # outputs "y 3" + ``` Control flow statements @@ -261,8 +277,7 @@ If statement The if statement is one way to branch the control flow: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" let name = readLine(stdin) if name == "": echo "Poor soul, you lost your name?" @@ -270,21 +285,21 @@ The if statement is one way to branch the control flow: echo "Very funny, your name is name." else: echo "Hi, ", name, "!" + ``` -There can be zero or more ``elif`` parts, and the ``else`` part is optional. -The keyword ``elif`` is short for ``else if``, and is useful to avoid -excessive indentation. (The ``""`` is the empty string. It contains no +There can be zero or more `elif` parts, and the `else` part is optional. +The keyword `elif` is short for `else if`, and is useful to avoid +excessive indentation. (The `""` is the empty string. It contains no characters.) Case statement -------------- -Another way to branch is provided by the case statement. A case statement is -a multi-branch: +Another way to branch is provided by the case statement. A case statement allows +for multiple branches: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" let name = readLine(stdin) case name of "": @@ -295,44 +310,47 @@ a multi-branch: echo "Cool name!" else: echo "Hi, ", name, "!" + ``` -As it can be seen, for an ``of`` branch a comma separated list of values is also +As it can be seen, for an `of` branch a comma-separated list of values is also allowed. -The case statement can deal with integers, other ordinal types and strings. +The case statement can deal with integers, other ordinal types, and strings. (What an ordinal type is will be explained soon.) For integers or other ordinal types value ranges are also possible: -.. code-block:: nim + ```nim # this statement will be explained later: - from strutils import parseInt + from std/strutils import parseInt echo "A number please: " let n = parseInt(readLine(stdin)) case n of 0..2, 4..7: echo "The number is in the set: {0, 1, 2, 4, 5, 6, 7}" of 3, 8: echo "The number is 3 or 8" + ``` -However, the above code does not compile: the reason is that you have to cover -every value that ``n`` may contain, but the code only handles the values -``0..8``. Since it is not very practical to list every other possible integer +However, the above code **does not compile**: the reason is that you have to cover +every value that `n` may contain, but the code only handles the values +`0..8`. Since it is not very practical to list every other possible integer (though it is possible thanks to the range notation), we fix this by telling the compiler that for every other value nothing should be done: -.. code-block:: nim + ```nim ... case n of 0..2, 4..7: echo "The number is in the set: {0, 1, 2, 4, 5, 6, 7}" of 3, 8: echo "The number is 3 or 8" else: discard + ``` -The empty `discard statement <#procedures-discard-statement>`_ is a *do +The empty [discard statement] is a *do nothing* statement. The compiler knows that a case statement with an else part cannot fail and thus the error disappears. Note that it is impossible to cover -all possible string values: that is why string cases always need an ``else`` +all possible string values: that is why string cases always need an `else` branch. -In general the case statement is used for subrange types or enumerations where +In general, the case statement is used for subrange types or enumerations where it is of great help that the compiler checks that you covered any possible value. @@ -342,15 +360,13 @@ While statement The while statement is a simple looping construct: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" echo "What's your name? " var name = readLine(stdin) while name == "": echo "Please tell me your name: " - name = readLine(stdin) - # no ``var``, because we do not declare a new variable here + name = readLine(stdin) # no `var`, because we do not declare a new variable here + ``` The example uses a while loop to keep asking the users for their name, as long as the user types in nothing (only presses RETURN). @@ -359,139 +375,147 @@ as the user types in nothing (only presses RETURN). For statement ------------- -The ``for`` statement is a construct to loop over any element an *iterator* -provides. The example uses the built-in `countup -<system.html#countup.i,T,T,Positive>`_ iterator: +The `for` statement is a construct to loop over any element an *iterator* +provides. The example uses the built-in [countup]( +system.html#countup.i,T,T,Positive) iterator: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" echo "Counting to ten: " for i in countup(1, 10): echo i # --> Outputs 1 2 3 4 5 6 7 8 9 10 on different lines + ``` -The variable ``i`` is implicitly declared by the -``for`` loop and has the type ``int``, because that is what `countup -<system.html#countup.i,T,T,Positive>`_ returns. ``i`` runs through the values -1, 2, .., 10. Each value is ``echo``-ed. This code does the same: +The variable `i` is implicitly declared by the +`for` loop and has the type `int`, because that is what [countup]( +system.html#countup.i,T,T,Positive) returns. `i` runs through the values +1, 2, .., 10. Each value is `echo`-ed. This code does the same: -.. code-block:: nim + ```nim echo "Counting to 10: " var i = 1 while i <= 10: echo i - inc(i) # increment i by 1 + inc i # increment i by 1 # --> Outputs 1 2 3 4 5 6 7 8 9 10 on different lines + ``` + +Since counting up occurs so often in programs, Nim also has a [..]( +system.html#...i,T,T) iterator that does the same: + + ```nim + for i in 1 .. 10: + ... + ``` Counting down can be achieved as easily (but is less often needed): -.. code-block:: nim + ```nim echo "Counting down from 10 to 1: " for i in countdown(10, 1): echo i # --> Outputs 10 9 8 7 6 5 4 3 2 1 on different lines + ``` -Since counting up occurs so often in programs, Nim also has a `.. -<system.html#...i,T,T>`_ iterator that does the same: - -.. code-block:: nim - for i in 1 .. 10: - ... - -Zero-indexed counting has two shortcuts ``..<`` and ``.. ^1`` -(`backwards index operator <system.html#^.t%2Cint>`_) to simplify +Zero-indexed counting has two shortcuts `..<` and `.. ^1` +([backward index operator](system.html#^.t%2Cint)) to simplify counting to one less than the higher index: -.. code-block:: nim + ```nim for i in 0 ..< 10: - ... # 0 .. 9 + ... # the same as 0 .. 9 + ``` or -.. code-block:: nim + ```nim var s = "some string" for i in 0 ..< s.len: ... + ``` or -.. code-block:: nim + ```nim var s = "some string" for idx, c in s[0 .. ^1]: - ... + ... # ^1 is the last element, ^2 would be one before it, and so on + ``` Other useful iterators for collections (like arrays and sequences) are -* ``items`` and ``mitems``, which provides immutable and mutable elements respectively, and -* ``pairs`` and ``mpairs`` which provides the element and an index number (immutable and mutable respectively) +* `items` and `mitems`, which provides immutable and mutable elements respectively, and +* `pairs` and `mpairs` which provides the element and an index number (immutable and mutable respectively) -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" for index, item in ["a","b"].pairs: echo item, " at index ", index # => a at index 0 # => b at index 1 + ``` Scopes and the block statement ------------------------------ + Control flow statements have a feature not covered yet: they open a -new scope. This means that in the following example, ``x`` is not accessible +new scope. This means that in the following example, `x` is not accessible outside the loop: -.. code-block:: nim - :test: "nim c $1" - :status: 1 + ```nim test = "nim c $1" status = 1 while false: var x = "hi" echo x # does not work + ``` A while (for) statement introduces an implicit block. Identifiers -are only visible within the block they have been declared. The ``block`` +are only visible within the block they have been declared. The `block` statement can be used to open a new block explicitly: -.. code-block:: nim - :test: "nim c $1" - :status: 1 + ```nim test = "nim c $1" status = 1 block myblock: var x = "hi" echo x # does not work either + ``` -The block's *label* (``myblock`` in the example) is optional. +The block's *label* (`myblock` in the example) is optional. Break statement --------------- -A block can be left prematurely with a ``break`` statement. The break statement -can leave a ``while``, ``for``, or a ``block`` statement. It leaves the + +A block can be left prematurely with a `break` statement. The break statement +can leave a `while`, `for`, or a `block` statement. It leaves the innermost construct, unless a label of a block is given: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" block myblock: echo "entering block" while true: echo "looping" break # leaves the loop, but not the block echo "still in block" + echo "outside the block" block myblock2: echo "entering block" while true: echo "looping" break myblock2 # leaves the block (and the loop) - echo "still in block" + echo "still in block" # it won't be printed + echo "outside the block" + ``` Continue statement ------------------ -Like in many other programming languages, a ``continue`` statement starts + +Like in many other programming languages, a `continue` statement starts the next iteration immediately: -.. code-block:: nim - :test: "nim c $1" - while true: - let x = readLine(stdin) - if x == "": continue - echo x + ```nim test = "nim c $1" + for i in 1 .. 5: + if i <= 3: continue + echo i # will only print 4 and 5 + ``` When statement @@ -499,9 +523,7 @@ When statement Example: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" when system.hostOS == "windows": echo "running on Windows!" elif system.hostOS == "linux": @@ -510,18 +532,19 @@ Example: echo "running on Mac OS X!" else: echo "unknown operating system" + ``` -The ``when`` statement is almost identical to the ``if`` statement, but with these +The `when` statement is almost identical to the `if` statement, but with these differences: * Each condition must be a constant expression since it is evaluated by the compiler. * The statements within a branch do not open a new scope. * The compiler checks the semantics and produces code *only* for the statements - that belong to the first condition that evaluates to ``true``. + that belong to the first condition that evaluates to `true`. -The ``when`` statement is useful for writing platform specific code, similar to -the ``#ifdef`` construct in the C programming language. +The `when` statement is useful for writing platform-specific code, similar to +the `#ifdef`:c: construct in the C programming language. Statements and indentation @@ -530,15 +553,15 @@ Statements and indentation Now that we covered the basic control flow statements, let's return to Nim indentation rules. -In Nim there is a distinction between *simple statements* and *complex +In Nim, there is a distinction between *simple statements* and *complex statements*. *Simple statements* cannot contain other statements: -Assignment, procedure calls or the ``return`` statement belong to the simple -statements. *Complex statements* like ``if``, ``when``, ``for``, ``while`` can +Assignment, procedure calls, or the `return` statement are all simple +statements. *Complex statements* like `if`, `when`, `for`, `while` can contain other statements. To avoid ambiguities, complex statements must always be indented, but single simple statements do not: -.. code-block:: nim - # no indentation needed for single assignment statement: + ```nim + # no indentation needed for single-assignment statement: if x: x = false # indentation needed for nested if statement: @@ -552,41 +575,43 @@ be indented, but single simple statements do not: if x: x = false y = false + ``` -*Expressions* are parts of a statement which usually result in a value. The -condition in an if statement is an example for an expression. Expressions can +*Expressions* are parts of a statement that usually result in a value. The +condition in an if statement is an example of an expression. Expressions can contain indentation at certain places for better readability: -.. code-block:: nim - + ```nim if thisIsaLongCondition() and thisIsAnotherLongCondition(1, 2, 3, 4): x = true + ``` As a rule of thumb, indentation within expressions is allowed after operators, an open parenthesis and after commas. -With parenthesis and semicolons ``(;)`` you can use statements where only +With parenthesis and semicolons `(;)` you can use statements where only an expression is allowed: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" # computes fac(4) at compile time: const fac4 = (var x = 1; for i in 1..4: x *= i; x) + ``` Procedures ========== -To define new commands like `echo <system.html#echo,varargs[typed,]>`_ -and `readLine <io.html#readLine,File>`_ in the examples, the concept of a -`procedure` is needed. (Some languages call them *methods* or *functions*.) -In Nim new procedures are defined with the ``proc`` keyword: +To define new commands like [echo](system.html#echo,varargs[typed,]) +and [readLine](syncio.html#readLine,File) in the examples, the concept of a +*procedure* is needed. You might be used to them being called *methods* or +*functions* in other languages, but Nim +[differentiates these concepts](tut1.html#procedures-funcs-and-methods). In +Nim, new procedures are defined with the `proc` keyword: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc yes(question: string): bool = echo question, " (y/n)" while true: @@ -599,69 +624,81 @@ In Nim new procedures are defined with the ``proc`` keyword: echo "I'm sorry Dave, I'm afraid I can't do that." else: echo "I think you know what the problem is just as well as I do." + ``` -This example shows a procedure named ``yes`` that asks the user a ``question`` +This example shows a procedure named `yes` that asks the user a `question` and returns true if they answered "yes" (or something similar) and returns -false if they answered "no" (or something similar). A ``return`` statement +false if they answered "no" (or something similar). A `return` statement leaves the procedure (and therefore the while loop) immediately. The -``(question: string): bool`` syntax describes that the procedure expects a -parameter named ``question`` of type ``string`` and returns a value of type -``bool``. The ``bool`` type is built-in: the only valid values for ``bool`` are -``true`` and ``false``. -The conditions in if or while statements must be of type ``bool``. +`(question: string): bool` syntax describes that the procedure expects a +parameter named `question` of type `string` and returns a value of type +`bool`. The `bool` type is built-in: the only valid values for `bool` are +`true` and `false`. +The conditions in if or while statements must be of type `bool`. -Some terminology: in the example ``question`` is called a (formal) *parameter*, -``"Should I..."`` is called an *argument* that is passed to this parameter. +Some terminology: in the example `question` is called a (formal) *parameter*, +`"Should I..."` is called an *argument* that is passed to this parameter. Result variable --------------- -A procedure that returns a value has an implicit ``result`` variable declared -that represents the return value. A ``return`` statement with no expression is a -shorthand for ``return result``. The ``result`` value is always returned -automatically at the end of a procedure if there is no ``return`` statement at + +A procedure that returns a value has an implicit `result` variable declared +that represents the return value. A `return` statement with no expression is +shorthand for `return result`. The `result` value is always returned +automatically at the end of a procedure if there is no `return` statement at the exit. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc sumTillNegative(x: varargs[int]): int = for i in x: if i < 0: return result = result + i - echo sumTillNegative() # echos 0 - echo sumTillNegative(3, 4, 5) # echos 12 - echo sumTillNegative(3, 4 , -1 , 6) # echos 7 + echo sumTillNegative() # echoes 0 + echo sumTillNegative(3, 4, 5) # echoes 12 + echo sumTillNegative(3, 4 , -1 , 6) # echoes 7 + ``` -The ``result`` variable is already implicitly declared at the start of the +The `result` variable is already implicitly declared at the start of the function, so declaring it again with 'var result', for example, would shadow it with a normal variable of the same name. The result variable is also already -initialised with the type's default value. Note that referential data types will -be ``nil`` at the start of the procedure, and thus may require manual -initialisation. +initialized with the type's default value. Note that referential data types will +be `nil` at the start of the procedure, and thus may require manual +initialization. +A procedure that does not have any `return` statement and does not use the +special `result` variable returns the value of its last expression. For example, +this procedure + + ```nim test = "nim c $1" + proc helloWorld(): string = + "Hello, World!" + ``` + +returns the string "Hello, World!". Parameters ---------- + Parameters are immutable in the procedure body. By default, their value cannot be changed because this allows the compiler to implement parameter passing in the most efficient way. If a mutable variable is needed inside the procedure, it has -to be declared with ``var`` in the procedure body. Shadowing the parameter name +to be declared with `var` in the procedure body. Shadowing the parameter name is possible, and actually an idiom: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc printSeq(s: seq, nprinted: int = -1) = var nprinted = if nprinted == -1: s.len else: min(nprinted, s.len) for i in 0 ..< nprinted: echo s[i] + ``` If the procedure needs to modify the argument for the -caller, a ``var`` parameter can be used: +caller, a `var` parameter can be used: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc divmod(a, b: int; res, remainder: var int) = res = a div b # integer division remainder = a mod b # integer modulo operation @@ -671,8 +708,9 @@ caller, a ``var`` parameter can be used: divmod(8, 5, x, y) # modifies x and y echo x echo y + ``` -In the example, ``res`` and ``remainder`` are `var parameters`. +In the example, `res` and `remainder` are `var parameters`. Var parameters can be modified by the procedure and the changes are visible to the caller. Note that the above example would better make use of a tuple as a return value instead of using var parameters. @@ -680,23 +718,25 @@ a tuple as a return value instead of using var parameters. Discard statement ----------------- + To call a procedure that returns a value just for its side effects and ignoring -its return value, a ``discard`` statement **must** be used. Nim does not +its return value, a `discard` statement **must** be used. Nim does not allow silently throwing away a return value: -.. code-block:: nim + ```nim discard yes("May I ask a pointless question?") + ``` The return value can be ignored implicitly if the called proc/iterator has -been declared with the ``discardable`` pragma: +been declared with the `discardable` pragma: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc p(x, y: int): int {.discardable.} = return x + y p(3, 4) # now valid + ``` Named arguments @@ -704,54 +744,59 @@ Named arguments Often a procedure has many parameters and it is not clear in which order the parameters appear. This is especially true for procedures that construct a -complex data type. Therefore the arguments to a procedure can be named, so +complex data type. Therefore, the arguments to a procedure can be named, so that it is clear which argument belongs to which parameter: -.. code-block:: nim + ```nim proc createWindow(x, y, width, height: int; title: string; show: bool): Window = ... var w = createWindow(show = true, title = "My Application", x = 0, y = 0, height = 600, width = 800) + ``` -Now that we use named arguments to call ``createWindow`` the argument order +Now that we use named arguments to call `createWindow` the argument order does not matter anymore. Mixing named arguments with ordered arguments is also possible, but not very readable: -.. code-block:: nim + ```nim var w = createWindow(0, 0, title = "My Application", height = 600, width = 800, true) + ``` The compiler checks that each parameter receives exactly one argument. Default values -------------- -To make the ``createWindow`` proc easier to use it should provide `default + +To make the `createWindow` proc easier to use it should provide `default values`; these are values that are used as arguments if the caller does not specify them: -.. code-block:: nim + ```nim proc createWindow(x = 0, y = 0, width = 500, height = 700, title = "unknown", show = true): Window = ... var w = createWindow(title = "My Application", height = 600, width = 800) + ``` -Now the call to ``createWindow`` only needs to set the values that differ +Now the call to `createWindow` only needs to set the values that differ from the defaults. Note that type inference works for parameters with default values; there is -no need to write ``title: string = "unknown"``, for example. +no need to write `title: string = "unknown"`, for example. Overloaded procedures --------------------- + Nim provides the ability to overload procedures similar to C++: -.. code-block:: nim + ```nim proc toString(x: int): string = result = if x < 0: "negative" @@ -765,48 +810,49 @@ Nim provides the ability to overload procedures similar to C++: assert toString(13) == "positive" # calls the toString(x: int) proc assert toString(true) == "yep" # calls the toString(x: bool) proc + ``` -(Note that ``toString`` is usually the `$ <dollars.html>`_ operator in -Nim.) The compiler chooses the most appropriate proc for the ``toString`` +(Note that `toString` is usually the [$](dollars.html) operator in +Nim.) The compiler chooses the most appropriate proc for the `toString` calls. How this overloading resolution algorithm works exactly is not -discussed here (it will be specified in the manual soon). However, it does -not lead to nasty surprises and is based on a quite simple unification -algorithm. Ambiguous calls are reported as errors. +discussed here -- see the manual for details. Ambiguous calls are reported as errors. Operators --------- -The Nim library makes heavy use of overloading - one reason for this is that -each operator like ``+`` is just an overloaded proc. The parser lets you -use operators in `infix notation` (``a + b``) or `prefix notation` (``+ a``). + +The Nim standard library makes heavy use of overloading - one reason for this is that +each operator like `+` is just an overloaded proc. The parser lets you +use operators in *infix notation* (`a + b`) or *prefix notation* (`+ a`). An infix operator always receives two arguments, a prefix operator always one. (Postfix operators are not possible, because this would be ambiguous: does -``a @ @ b`` mean ``(a) @ (@b)`` or ``(a@) @ (b)``? It always means -``(a) @ (@b)``, because there are no postfix operators in Nim.) +`a @ @ b` mean `(a) @ (@b)` or `(a@) @ (b)`? It always means +`(a) @ (@b)`, because there are no postfix operators in Nim.) -Apart from a few built-in keyword operators such as ``and``, ``or``, ``not``, +Apart from a few built-in keyword operators such as `and`, `or`, `not`, operators always consist of these characters: -``+ - * \ / < > = @ $ ~ & % ! ? ^ . |`` +`+ - * \ / < > = @ $ ~ & % ! ? ^ . |` -User defined operators are allowed. Nothing stops you from defining your own -``@!?+~`` operator, but doing so may reduce readability. +User-defined operators are allowed. Nothing stops you from defining your own +`@!?+~` operator, but doing so may reduce readability. The operator's precedence is determined by its first character. The details -can be found in the manual. +can be [found in the manual](manual.html#syntax-precedence). -To define a new operator enclose the operator in backticks "``": +To define a new operator enclose the operator in backticks "`": -.. code-block:: nim + ```nim proc `$` (x: myDataType): string = ... # now the $ operator also works with myDataType, overloading resolution # ensures that $ works for built-in types just like before + ``` -The "``" notation can also be used to call an operator just like any other +The "`" notation can also be used to call an operator just like any other procedure: -.. code-block:: nim - :test: "nim c $1" - if `==`( `+`(3, 4), 7): echo "True" + ```nim test = "nim c $1" + if `==`( `+`(3, 4), 7): echo "true" + ``` Forward declarations @@ -814,14 +860,15 @@ Forward declarations Every variable, procedure, etc. needs to be declared before it can be used. (The reason for this is that it is non-trivial to avoid this need in a -language that supports meta programming as extensively as Nim does.) +language that supports metaprogramming as extensively as Nim does.) However, this cannot be done for mutually recursive procedures: -.. code-block:: nim + ```nim # forward declaration: proc even(n: int): bool + ``` -.. code-block:: nim + ```nim proc odd(n: int): bool = assert(n >= 0) # makes sure we don't run into negative recursion if n == 0: false @@ -833,12 +880,13 @@ However, this cannot be done for mutually recursive procedures: if n == 1: false else: n == 0 or odd(n-1) + ``` -Here ``odd`` depends on ``even`` and vice versa. Thus ``even`` needs to be +Here `odd` depends on `even` and vice versa. Thus `even` needs to be introduced to the compiler before it is completely defined. The syntax for -such a forward declaration is simple: just omit the ``=`` and the -procedure's body. The ``assert`` just adds border conditions, and will be -covered later in `Modules`_ section. +such a forward declaration is simple: just omit the `=` and the +procedure's body. The `assert` just adds border conditions, and will be +covered later in [Modules] section. Later versions of the language will weaken the requirements for forward declarations. @@ -847,60 +895,86 @@ The example also shows that a proc's body can consist of a single expression whose value is then returned implicitly. +Funcs and methods +----------------- + +As mentioned in the introduction, Nim differentiates between procedures, +functions, and methods, defined by the `proc`, `func`, and `method` keywords +respectively. In some ways, Nim is a bit more pedantic in its definitions than +other languages. + +Functions are closer to the concept of a pure mathematical +function, which might be familiar to you if you've ever done functional +programming. Essentially they are procedures with additional limitations set on +them: they can't access global state (except `const`) and can't produce +side-effects. The `func` keyword is basically an alias for `proc` tagged +with `{.noSideEffects.}`. Functions can still change their mutable arguments +however, which are those marked as `var`, along with any `ref` objects. + +Unlike procedures, methods are dynamically dispatched. This sounds a bit +complicated, but it is a concept closely related to inheritance and object-oriented +programming. If you overload a procedure (two procedures with the same name but +of different types or with different sets of arguments are said to be overloaded), the procedure to use is determined +at compile-time. Methods, on the other hand, depend on objects that inherit from +the `RootObj`. This is something that is covered in much greater depth in +the [second part of the tutorial](tut2.html#object-oriented-programming-dynamic-dispatch). + + Iterators ========= Let's return to the simple counting example: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" echo "Counting to ten: " for i in countup(1, 10): echo i + ``` -Can a `countup <system.html#countup.i,T,T,Positive>`_ proc be written that -supports this loop? Lets try: +Can a [countup](system.html#countup.i,T,T,Positive) proc be written that +supports this loop? Let's try: -.. code-block:: nim + ```nim proc countup(a, b: int): int = var res = a while res <= b: return res inc(res) + ``` However, this does not work. The problem is that the procedure should not -only ``return``, but return and **continue** after an iteration has +only `return`, but return and **continue** after an iteration has finished. This *return and continue* is called a `yield` statement. Now -the only thing left to do is to replace the ``proc`` keyword by ``iterator`` -and here it is - our first iterator: +the only thing left to do is to replace the `proc` keyword by `iterator` +and here it is -- our first iterator: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" iterator countup(a, b: int): int = var res = a while res <= b: yield res inc(res) + ``` Iterators look very similar to procedures, but there are several important differences: * Iterators can only be called from for loops. -* Iterators cannot contain a ``return`` statement (and procs cannot contain a - ``yield`` statement). -* Iterators have no implicit ``result`` variable. +* Iterators cannot contain a `return` statement (and procs cannot contain a + `yield` statement). +* Iterators have no implicit `result` variable. * Iterators do not support recursion. -* Iterators cannot be forward declared, because the compiler must be able - to inline an iterator. (This restriction will be gone in a - future version of the compiler.) - -However, you can also use a ``closure`` iterator to get a different set of -restrictions. See `first class iterators <manual.html#iterators-and-the-for-statement-first-class-iterators>`_ -for details. Iterators can have the same name and parameters as a proc, since -essentially they have their own namespaces. Therefore it is common practice to +* Iterators cannot be forward declared, because the compiler must be able to inline an iterator. + (This restriction will be gone in a future version of the compiler.) + +However, you can also use a closure iterator to get a different set of +restrictions. See [first-class iterators]( +manual.html#iterators-and-the-for-statement-firstminusclass-iterators) +for details. Iterators can have the same name and parameters as a proc since +essentially they have their own namespaces. Therefore, it is common to wrap iterators in procs of the same name which accumulate the result of the -iterator and return it as a sequence, like ``split`` from the `strutils module -<strutils.html>`_. +iterator and return it as a sequence, like `split` from the [strutils module]( +strutils.html). Basic types @@ -912,82 +986,85 @@ that are available for them in detail. Booleans -------- -Nim's boolean type is called ``bool`` and consists of the two -pre-defined values ``true`` and ``false``. Conditions in while, -if, elif, and when statements must be of type bool. +Nim's boolean type is called `bool` and consists of the two +pre-defined values `true` and `false`. Conditions in `while`, +`if`, `elif`, and `when` statements must be of type bool. -The operators ``not, and, or, xor, <, <=, >, >=, !=, ==`` are defined -for the bool type. The ``and`` and ``or`` operators perform short-circuit +The operators `not, and, or, xor, <, <=, >, >=, !=, ==` are defined +for the bool type. The `and` and `or` operators perform short-circuit evaluation. For example: -.. code-block:: nim - + ```nim while p != nil and p.name != "xyz": # p.name is not evaluated if p == nil p = p.next + ``` Characters ---------- -The `character type` is called ``char``. Its size is always one byte, so -it cannot represent most UTF-8 characters; but it *can* represent one of the bytes -that makes up a multi-byte UTF-8 character. + +The *character type* is called `char`. Its size is always one byte, so +it cannot represent most UTF-8 characters, but it *can* represent one of the bytes +that makes up a multibyte UTF-8 character. The reason for this is efficiency: for the overwhelming majority of use-cases, -the resulting programs will still handle UTF-8 properly as UTF-8 was specially +the resulting programs will still handle UTF-8 properly as UTF-8 was especially designed for this. Character literals are enclosed in single quotes. -Chars can be compared with the ``==``, ``<``, ``<=``, ``>``, ``>=`` operators. -The ``$`` operator converts a ``char`` to a ``string``. Chars cannot be mixed -with integers; to get the ordinal value of a ``char`` use the ``ord`` proc. -Converting from an integer to a ``char`` is done with the ``chr`` proc. +Chars can be compared with the `==`, `<`, `<=`, `>`, `>=` operators. +The `$` operator converts a `char` to a `string`. Chars cannot be mixed +with integers; to get the ordinal value of a `char` use the `ord` proc. +Converting from an integer to a `char` is done with the `chr` proc. Strings ------- + String variables are **mutable**, so appending to a string is possible, and quite efficient. Strings in Nim are both zero-terminated and have a -length field. A string's length can be retrieved with the builtin ``len`` +length field. A string's length can be retrieved with the builtin `len` procedure; the length never counts the terminating zero. Accessing the terminating zero is an error, it only exists so that a Nim string can be converted -to a ``cstring`` without doing a copy. +to a `cstring` without doing a copy. -The assignment operator for strings copies the string. You can use the ``&`` -operator to concatenate strings and ``add`` to append to a string. +The assignment operator for strings copies the string. You can use the `&` +operator to concatenate strings and `add` to append to a string. Strings are compared using their lexicographical order. All the comparison operators are supported. By convention, all strings are UTF-8 encoded, but this is not enforced. For example, when reading strings from binary files, they are merely -a sequence of bytes. The index operation ``s[i]`` means the i-th *char* of -``s``, not the i-th *unichar*. +a sequence of bytes. The index operation `s[i]` means the i-th *char* of +`s`, not the i-th *unichar*. -A string variable is initialized with the empty string ``""``. +A string variable is initialized with the empty string `""`. Integers -------- + Nim has these integer types built-in: -``int int8 int16 int32 int64 uint uint8 uint16 uint32 uint64``. +`int int8 int16 int32 int64 uint uint8 uint16 uint32 uint64`. -The default integer type is ``int``. Integer literals can have a *type suffix* +The default integer type is `int`. Integer literals can have a *type suffix* to specify a non-default integer type: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" let - x = 0 # x is of type ``int`` - y = 0'i8 # y is of type ``int8`` - z = 0'i64 # z is of type ``int64`` - u = 0'u # u is of type ``uint`` + x = 0 # x is of type `int` + y = 0'i8 # y is of type `int8` + z = 0'i32 # z is of type `int32` + u = 0'u # u is of type `uint` + ``` Most often integers are used for counting objects that reside in memory, so -``int`` has the same size as a pointer. +`int` has the same size as a pointer. -The common operators ``+ - * div mod < <= == != > >=`` are defined for -integers. The ``and or xor not`` operators are also defined for integers, and -provide *bitwise* operations. Left bit shifting is done with the ``shl``, right -shifting with the ``shr`` operator. Bit shifting operators always treat their +The common operators `+ - * div mod < <= == != > >=` are defined for +integers. The `and or xor not` operators are also defined for integers and +provide *bitwise* operations. Left bit shifting is done with the `shl`, right +shifting with the `shr` operator. Bit shifting operators always treat their arguments as *unsigned*. For `arithmetic bit shifts`:idx: ordinary multiplication or division can be used. @@ -996,66 +1073,66 @@ errors. Lossless `Automatic type conversion`:idx: is performed in expressions where different kinds of integer types are used. However, if the type conversion -would cause loss of information, the `EOutOfRange`:idx: exception is raised (if the error +would cause loss of information, the `RangeDefect`:idx: is raised (if the error cannot be detected at compile time). Floats ------ -Nim has these floating point types built-in: ``float float32 float64``. -The default float type is ``float``. In the current implementation, -``float`` is always 64-bits. +Nim has these floating-point types built-in: `float float32 float64`. + +The default float type is `float`. In the current implementation, +`float` is always 64-bits. Float literals can have a *type suffix* to specify a non-default float type: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" var - x = 0.0 # x is of type ``float`` - y = 0.0'f32 # y is of type ``float32`` - z = 0.0'f64 # z is of type ``float64`` + x = 0.0 # x is of type `float` + y = 0.0'f32 # y is of type `float32` + z = 0.0'f64 # z is of type `float64` + ``` -The common operators ``+ - * / < <= == != > >=`` are defined for +The common operators `+ - * / < <= == != > >=` are defined for floats and follow the IEEE-754 standard. -Automatic type conversion in expressions with different kinds of floating -point types is performed: the smaller type is converted to the larger. Integer -types are **not** converted to floating point types automatically, nor vice -versa. Use the `toInt <system.html#toInt,float>`_ and -`toFloat <system.html#toFloat,int>`_ procs for these conversions. +Automatic type conversion in expressions with different kinds of floating-point types is performed: the smaller type is converted to the larger. Integer +types are **not** converted to floating-point types automatically, nor vice +versa. Use the [toInt](system.html#toInt,float) and +[toFloat](system.html#toFloat,int) procs for these conversions. Type Conversion --------------- + Conversion between numerical types is performed by using the type as a function: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" var x: int32 = 1.int32 # same as calling int32(1) y: int8 = int8('a') # 'a' == 97'i8 z: float = 2.5 # int(2.5) rounds down to 2 sum: int = int(x) + int(y) + int(z) # sum == 100 + ``` Internal type representation ============================ -As mentioned earlier, the built-in `$ <dollars.html>`_ (stringify) operator +As mentioned earlier, the built-in [$](dollars.html) (stringify) operator turns any basic type into a string, which you can then print to the console -using the ``echo`` proc. However, advanced types, and your own custom types, -won't work with the ``$`` operator until you define it for them. +using the `echo` proc. However, advanced types, and your own custom types, +won't work with the `$` operator until you define it for them. Sometimes you just want to debug the current value of a complex type without -having to write its ``$`` operator. You can use then the `repr -<system.html#repr,T>`_ proc which works with any type and even complex data +having to write its `$` operator. You can use then the [repr]( +system.html#repr,T) proc which works with any type and even complex data graphs with cycles. The following example shows that even for basic types -there is a difference between the ``$`` and ``repr`` outputs: +there is a difference between the `$` and `repr` outputs: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" var myBool = true myCharacter = 'n' @@ -1071,47 +1148,48 @@ there is a difference between the ``$`` and ``repr`` outputs: echo myInteger, ":", repr(myInteger) # --> 42:42 echo myFloat, ":", repr(myFloat) - # --> 3.1400000000000001e+00:3.1400000000000001e+00 + # --> 3.14:3.14 + ``` Advanced types ============== -In Nim new types can be defined within a ``type`` statement: +In Nim new types can be defined within a `type` statement: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type biggestInt = int64 # biggest integer type that is available biggestFloat = float64 # biggest float type that is available + ``` Enumeration and object types may only be defined within a -``type`` statement. +`type` statement. Enumerations ------------ + A variable of an enumeration type can only be assigned one of the enumeration's specified values. These values are a set of ordered symbols. Each symbol is mapped to an integer value internally. The first symbol is represented -at runtime by 0, the second by 1 and so on. For example: - -.. code-block:: nim - :test: "nim c $1" +at runtime by 0, the second by 1, and so on. For example: + ```nim test = "nim c $1" type Direction = enum north, east, south, west var x = south # `x` is of type `Direction`; its value is `south` - echo x # writes "south" to `stdout` + echo x # prints "south" + ``` All the comparison operators can be used with enumeration types. An enumeration's symbol can be qualified to avoid ambiguities: -``Direction.south``. +`Direction.south`. -The ``$`` operator can convert any enumeration value to its name, and the ``ord`` +The `$` operator can convert any enumeration value to its name, and the `ord` proc can convert it to its underlying integer value. For better interfacing to other programming languages, the symbols of enum @@ -1121,53 +1199,57 @@ must be in ascending order. Ordinal types ------------- -Enumerations, integer types, ``char`` and ``bool`` (and + +Enumerations, integer types, `char` and `bool` (and subranges) are called ordinal types. Ordinal types have quite a few special operations: ------------------ -------------------------------------------------------- + +================= ======================================================== Operation Comment ------------------ -------------------------------------------------------- -``ord(x)`` returns the integer value that is used to +================= ======================================================== +`ord(x)` returns the integer value that is used to represent `x`'s value -``inc(x)`` increments `x` by one -``inc(x, n)`` increments `x` by `n`; `n` is an integer -``dec(x)`` decrements `x` by one -``dec(x, n)`` decrements `x` by `n`; `n` is an integer -``succ(x)`` returns the successor of `x` -``succ(x, n)`` returns the `n`'th successor of `x` -``pred(x)`` returns the predecessor of `x` -``pred(x, n)`` returns the `n`'th predecessor of `x` ------------------ -------------------------------------------------------- - -The `inc <system.html#inc,T,int>`_, `dec <system.html#dec,T,int>`_, `succ -<system.html#succ,T,int>`_ and `pred <system.html#pred,T,int>`_ operations can -fail by raising an `EOutOfRange` or `EOverflow` exception. (If the code has been +`inc(x)` increments `x` by one +`inc(x, n)` increments `x` by `n`; `n` is an integer +`dec(x)` decrements `x` by one +`dec(x, n)` decrements `x` by `n`; `n` is an integer +`succ(x)` returns the successor of `x` +`succ(x, n)` returns the `n`'th successor of `x` +`pred(x)` returns the predecessor of `x` +`pred(x, n)` returns the `n`'th predecessor of `x` +================= ======================================================== + + +The [inc](system.html#inc,T,int), [dec](system.html#dec,T,int), [succ]( +system.html#succ,T,int) and [pred](system.html#pred,T,int) operations can +fail by raising an `RangeDefect` or `OverflowDefect`. (If the code has been compiled with the proper runtime checks turned on.) Subranges --------- + A subrange type is a range of values from an integer or enumeration type (the base type). Example: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type MySubrange = range[0..5] + ``` -``MySubrange`` is a subrange of ``int`` which can only hold the values 0 -to 5. Assigning any other value to a variable of type ``MySubrange`` is a +`MySubrange` is a subrange of `int` which can only hold the values 0 +to 5. Assigning any other value to a variable of type `MySubrange` is a compile-time or runtime error. Assignments from the base type to one of its subrange types (and vice versa) are allowed. -The ``system`` module defines the important `Natural <system.html#Natural>`_ -type as ``range[0..high(int)]`` (`high <system.html#high,typedesc[T]>`_ returns +The `system` module defines the important [Natural](system.html#Natural) +type as `range[0..high(int)]` ([high](system.html#high,typedesc[T]) returns the maximal value). Other programming languages may suggest the use of unsigned integers for natural numbers. This is often **unwise**: you don't want unsigned arithmetic (which wraps around) just because the numbers cannot be negative. -Nim's ``Natural`` type helps to avoid this common programming error. +Nim's `Natural` type helps to avoid this common programming error. Sets @@ -1177,23 +1259,23 @@ Sets Arrays ------ -An array is a simple fixed length container. Each element in -an array has the same type. The array's index type can be any ordinal type. -Arrays can be constructed using ``[]``: +An array is a simple fixed-length container. Each element in +an array has the same type. The array's index type can be any ordinal type. -.. code-block:: nim - :test: "nim c $1" +Arrays can be constructed using `[]`: + ```nim test = "nim c $1" type IntArray = array[0..5, int] # an array that is indexed with 0..5 var x: IntArray x = [1, 2, 3, 4, 5, 6] - for i in low(x)..high(x): + for i in low(x) .. high(x): echo x[i] + ``` -The notation ``x[i]`` is used to access the i-th element of ``x``. +The notation `x[i]` is used to access the i-th element of `x`. Array access is always bounds checked (at compile-time or at runtime). These checks can be disabled via pragmas or invoking the compiler with the ``--bound_checks:off`` command line switch. @@ -1201,13 +1283,12 @@ checks can be disabled via pragmas or invoking the compiler with the Arrays are value types, like any other Nim type. The assignment operator copies the whole array contents. -The built-in `len <system.html#len,TOpenArray>`_ proc returns the array's -length. `low(a) <system.html#low,openArray[T]>`_ returns the lowest valid index -for the array `a` and `high(a) <system.html#high,openArray[T]>`_ the highest +The built-in [len](system.html#len,TOpenArray) proc returns the array's +length. [low(a)](system.html#low,openArray[T]) returns the lowest valid index +for the array `a` and [high(a)](system.html#high,openArray[T]) the highest valid index. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type Direction = enum north, east, south, west @@ -1219,10 +1300,11 @@ valid index. level[north] = on level[south] = slowBlink level[east] = fastBlink - echo repr(level) # --> [on, fastBlink, slowBlink, off] + echo level # --> [on, fastBlink, slowBlink, off] echo low(level) # --> north echo len(level) # --> 4 echo high(level) # --> west + ``` The syntax for nested arrays (multidimensional) in other languages is a matter of appending more brackets because usually each dimension is restricted to the @@ -1230,9 +1312,9 @@ same index type as the others. In Nim you can have different dimensions with different index types, so the nesting syntax is slightly different. Building on the previous example where a level is defined as an array of enums indexed by yet another enum, we can add the following lines to add a light tower type -subdivided in height levels accessed through their integer index: +subdivided into height levels accessed through their integer index: -.. code-block:: nim + ```nim type LightTower = array[1..10, LevelSetting] var @@ -1241,25 +1323,26 @@ subdivided in height levels accessed through their integer index: tower[1][east] = mediumBlink echo len(tower) # --> 10 echo len(tower[1]) # --> 4 - echo repr(tower) # --> [[slowBlink, mediumBlink, ...more output.. + echo tower # --> [[slowBlink, mediumBlink, ...more output.. # The following lines don't compile due to type mismatch errors #tower[north][east] = on #tower[0][1] = on + ``` -Note how the built-in ``len`` proc returns only the array's first dimension -length. Another way of defining the ``LightTower`` to better illustrate its -nested nature would be to omit the previous definition of the ``LevelSetting`` +Note how the built-in `len` proc returns only the array's first dimension +length. Another way of defining the `LightTower` to better illustrate its +nested nature would be to omit the previous definition of the `LevelSetting` type and instead write it embedded directly as the type of the first dimension: -.. code-block:: nim + ```nim type LightTower = array[1..10, array[north..west, BlinkLights]] + ``` It is quite common to have arrays start at zero, so there's a shortcut syntax to specify a range from zero to the specified index minus one: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type IntArray = array[0..5, int] # an array that is indexed with 0..5 QuickArray = array[6, int] # an array that is indexed with 0..5 @@ -1268,50 +1351,50 @@ to specify a range from zero to the specified index minus one: y: QuickArray x = [1, 2, 3, 4, 5, 6] y = x - for i in low(x)..high(x): + for i in low(x) .. high(x): echo x[i], y[i] + ``` Sequences --------- + Sequences are similar to arrays but of dynamic length which may change during runtime (like strings). Since sequences are resizable they are always allocated on the heap and garbage collected. -Sequences are always indexed with an ``int`` starting at position 0. The `len -<system.html#len,seq[T]>`_, `low <system.html#low,openArray[T]>`_ and `high -<system.html#high,openArray[T]>`_ operations are available for sequences too. -The notation ``x[i]`` can be used to access the i-th element of ``x``. +Sequences are always indexed with an `int` starting at position 0. The [len]( +system.html#len,seq[T]), [low](system.html#low,openArray[T]) and [high]( +system.html#high,openArray[T]) operations are available for sequences too. +The notation `x[i]` can be used to access the i-th element of `x`. -Sequences can be constructed by the array constructor ``[]`` in conjunction -with the array to sequence operator ``@``. Another way to allocate space for -a sequence is to call the built-in `newSeq <system.html#newSeq>`_ procedure. +Sequences can be constructed by the array constructor `[]` in conjunction +with the array to sequence operator `@`. Another way to allocate space for +a sequence is to call the built-in [newSeq](system.html#newSeq) procedure. A sequence may be passed to an openarray parameter. Example: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" var x: seq[int] # a reference to a sequence of integers x = @[1, 2, 3, 4, 5, 6] # the @ turns the array into a sequence allocated on the heap + ``` -Sequence variables are initialized with ``@[]``. +Sequence variables are initialized with `@[]`. -The ``for`` statement can be used with one or two variables when used with a +The `for` statement can be used with one or two variables when used with a sequence. When you use the one variable form, the variable will hold the value -provided by the sequence. The ``for`` statement is looping over the results -from the `items() <system.html#items.i,seq[T]>`_ iterator from the `system -<system.html>`_ module. But if you use the two variable form, the first +provided by the sequence. The `for` statement is looping over the results +from the [items()](iterators.html#items.i,seq[T]) iterator from the [system]( +system.html) module. But if you use the two-variable form, the first variable will hold the index position and the second variable will hold the -value. Here the ``for`` statement is looping over the results from the -`pairs() <system.html#pairs.i,seq[T]>`_ iterator from the `system -<system.html>`_ module. Examples: +value. Here the `for` statement is looping over the results from the +[pairs()](iterators.html#pairs.i,seq[T]) iterator from the [system]( +system.html) module. Examples: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" for value in @[3, 4, 5]: echo value # --> 3 @@ -1323,22 +1406,23 @@ value. Here the ``for`` statement is looping over the results from the # --> index: 0, value:3 # --> index: 1, value:4 # --> index: 2, value:5 + ``` Open arrays ----------- + **Note**: Openarrays can only be used for parameters. -Often fixed size arrays turn out to be too inflexible; procedures should be +Often fixed-size arrays turn out to be too inflexible; procedures should be able to deal with arrays of different sizes. The `openarray`:idx: type allows -this. Openarrays are always indexed with an ``int`` starting at position 0. -The `len <system.html#len,TOpenArray>`_, `low <system.html#low,openArray[T]>`_ -and `high <system.html#high,openArray[T]>`_ operations are available for open +this. Openarrays are always indexed with an `int` starting at position 0. +The [len](system.html#len,TOpenArray), [low](system.html#low,openArray[T]) +and [high](system.html#high,openArray[T]) operations are available for open arrays too. Any array with a compatible base type can be passed to an openarray parameter, the index type does not matter. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" var fruits: seq[string] # reference to a sequence of strings that is initialized with '@[]' capitals: array[3, string] # array of strings with a fixed size @@ -1352,6 +1436,7 @@ openarray parameter, the index type does not matter. assert openArraySize(fruits) == 2 # procedure accepts a sequence as parameter assert openArraySize(capitals) == 3 # but also an array type + ``` The openarray type cannot be nested: multidimensional openarrays are not supported because this is seldom needed and cannot be done efficiently. @@ -1360,13 +1445,12 @@ supported because this is seldom needed and cannot be done efficiently. Varargs ------- -A ``varargs`` parameter is like an openarray parameter. However, it is +A `varargs` parameter is like an openarray parameter. However, it is also a means to implement passing a variable number of arguments to a procedure. The compiler converts the list of arguments to an array automatically: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc myWriteln(f: File, a: varargs[string]) = for s in items(a): write(f, s) @@ -1375,13 +1459,13 @@ to an array automatically: myWriteln(stdout, "abc", "def", "xyz") # is transformed by the compiler to: myWriteln(stdout, ["abc", "def", "xyz"]) + ``` This transformation is only done if the varargs parameter is the last parameter in the procedure header. It is also possible to perform type conversions in this context: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" proc myWriteln(f: File, a: varargs[string, `$`]) = for s in items(a): write(f, s) @@ -1390,9 +1474,10 @@ type conversions in this context: myWriteln(stdout, 123, "abc", 4.0) # is transformed by the compiler to: myWriteln(stdout, [$123, $"abc", $4.0]) + ``` -In this example `$ <dollars.html>`_ is applied to any argument that is passed -to the parameter ``a``. Note that `$ <dollars.html>`_ applied to strings is a +In this example [$](dollars.html) is applied to any argument that is passed +to the parameter `a`. Note that [$](dollars.html) applied to strings is a nop. @@ -1404,9 +1489,7 @@ context. A slice is just an object of type Slice which contains two bounds, `a` and `b`. By itself a slice is not very useful, but other collection types define operators which accept Slice objects to define ranges. -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" var a = "Nim is a programming language" b = "Slices are useless." @@ -1414,42 +1497,43 @@ define operators which accept Slice objects to define ranges. echo a[7 .. 12] # --> 'a prog' b[11 .. ^2] = "useful" echo b # --> 'Slices are useful.' + ``` In the previous example slices are used to modify a part of a string. The slice's bounds can hold any value supported by their type, but it is the proc using the slice object which defines what values are accepted. -To understand some of the different ways of specifying the indices of +To understand the different ways of specifying the indices of strings, arrays, sequences, etc., it must be remembered that Nim uses zero-based indices. -So the string ``b`` is of length 19, and two different ways of specifying the +So the string `b` is of length 19, and two different ways of specifying the indices are -.. code-block:: nim - + ```nim "Slices are useless." | | | 0 11 17 using indices ^19 ^8 ^2 using ^ syntax + ``` -where ``b[0 .. ^1]`` is equivalent to ``b[0 .. b.len-1]`` and ``b[0 ..< b.len]``, and it -can be seen that the ``^1`` provides a short-hand way of specifying the ``b.len-1``. See -the `backwards index operator <system.html#^.t%2Cint>`_. +where `b[0 .. ^1]` is equivalent to `b[0 .. b.len-1]` and `b[0 ..< b.len]`, and it +can be seen that the `^1` provides a shorthand way of specifying the `b.len-1`. See +the [backwards index operator](system.html#^.t%2Cint). In the above example, because the string ends in a period, to get the portion of the string that is "useless" and replace it with "useful". -``b[11 .. ^2]`` is the portion "useless", and ``b[11 .. ^2] = "useful"`` replaces the +`b[11 .. ^2]` is the portion "useless", and `b[11 .. ^2] = "useful"` replaces the "useless" portion with "useful", giving the result "Slices are useful." -Note 1: alternate ways of writing this are ``b[^8 .. ^2] = "useful"`` or -as ``b[11 .. b.len-2] = "useful"`` or as ``b[11 ..< b.len-1] = "useful"``. +Note 1: alternate ways of writing this are `b[^8 .. ^2] = "useful"` or +as `b[11 .. b.len-2] = "useful"` or as `b[11 ..< b.len-1] = "useful"`. -Note 2: As the ``^`` template returns a `distinct int <manual.html#types-distinct-type>`_ -of type ``BackwardsIndex``, we can have a ``lastIndex`` constant defined as ``const lastIndex = ^1``, -and later used as ``b[0 .. lastIndex]``. +Note 2: As the `^` template returns a [distinct int](manual.html#types-distinct-type) +of type `BackwardsIndex`, we can have a `lastIndex` constant defined as `const lastIndex = ^1`, +and later used as `b[0 .. lastIndex]`. Objects ------- @@ -1459,11 +1543,11 @@ structure with a name is the object type. An object is a value type, which means that when an object is assigned to a new variable all its components are copied as well. -Each object type ``Foo`` has a constructor ``Foo(field: value, ...)`` +Each object type `Foo` has a constructor `Foo(field: value, ...)` where all of its fields can be initialized. Unspecified fields will get their default value. -.. code-block:: nim + ```nim type Person = object name: string @@ -1490,18 +1574,18 @@ get their default value. # unspecified members will be initialized with their default # values. In this case it is the empty string. doAssert person4.name == "" + ``` Object fields that should be visible from outside the defining module have to -be marked with ``*``. - -.. code-block:: nim - :test: "nim c $1" +be marked with `*`. + ```nim test = "nim c $1" type Person* = object # the type is visible from other modules name*: string # the field of this type is visible from other modules age*: int + ``` Tuples ------ @@ -1512,19 +1596,18 @@ Unlike object types though, tuple types are structurally typed, meaning different tuple-types are *equivalent* if they specify fields of the same type and of the same name in the same order. -The constructor ``()`` can be used to construct tuples. The order of the +The constructor `()` can be used to construct tuples. The order of the fields in the constructor must match the order in the tuple's definition. But unlike objects, a name for the tuple type may not be used here. -Like the object type the notation ``t.field`` is used to access a +Like the object type the notation `t.field` is used to access a tuple's field. Another notation that is not available for objects is -``t[i]`` to access the ``i``'th field. Here ``i`` must be a constant +`t[i]` to access the `i`'th field. Here `i` must be a constant integer. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type # type representing a person: # A person consists of a name and an age. @@ -1573,127 +1656,153 @@ integer. #person = building # --> Error: type mismatch: got (tuple[street: string, number: int]) # but expected 'Person' + ``` Even though you don't need to declare a type for a tuple to use it, tuples created with different field names will be considered different objects despite having the same field types. -Tuples can be *unpacked* during variable assignment (and only then!). This can +Tuples can be *unpacked* during variable assignment. This can be handy to assign directly the fields of the tuples to individually named -variables. An example of this is the `splitFile <os.html#splitFile,string>`_ -proc from the `os module <os.html>`_ which returns the directory, name and +variables. An example of this is the [splitFile](os.html#splitFile,string) +proc from the [os module](os.html) which returns the directory, name, and extension of a path at the same time. For tuple unpacking to work you must use parentheses around the values you want to assign the unpacking to, -otherwise you will be assigning the same value to all the individual +otherwise, you will be assigning the same value to all the individual variables! For example: -.. code-block:: nim - :test: "nim c $1" - - import os + ```nim test = "nim c $1" + import std/os let path = "usr/local/nimc.html" (dir, name, ext) = splitFile(path) baddir, badname, badext = splitFile(path) - echo dir # outputs `usr/local` - echo name # outputs `nimc` - echo ext # outputs `.html` + echo dir # outputs "usr/local" + echo name # outputs "nimc" + echo ext # outputs ".html" # All the following output the same line: - # `(dir: usr/local, name: nimc, ext: .html)` + # "(dir: usr/local, name: nimc, ext: .html)" echo baddir echo badname echo badext + ``` + +Tuple unpacking is also supported in for-loops: + + ```nim test = "nim c $1" + let a = [(10, 'a'), (20, 'b'), (30, 'c')] + + for (x, c) in a: + echo x + # This will output: 10; 20; 30 -Fields of tuples are always public, they don't need to be explicity + # Accessing the index is also possible: + for i, (x, c) in a: + echo i, c + # This will output: 0a; 1b; 2c + ``` + +Fields of tuples are always public, they don't need to be explicitly marked to be exported, unlike for example fields in an object type. + Reference and pointer types --------------------------- + References (similar to pointers in other programming languages) are a way to introduce many-to-one relationships. This means different references can point to and modify the same location in memory. Nim distinguishes between `traced`:idx: and `untraced`:idx: references. Untraced references are also called *pointers*. Traced references point to -objects in a garbage collected heap, untraced references point to -manually allocated objects or to objects elsewhere in memory. Thus -untraced references are *unsafe*. However for certain low-level operations -(e.g., accessing the hardware), untraced references are necessary. +objects in a garbage-collected heap, untraced references point to +manually allocated objects or objects elsewhere in memory. Thus, +untraced references are *unsafe*. However, for certain low-level operations +(e.g. accessing the hardware), untraced references are necessary. Traced references are declared with the **ref** keyword; untraced references are declared with the **ptr** keyword. -The empty ``[]`` subscript notation can be used to *derefer* a reference, -meaning to retrieve the item the reference points to. The ``.`` (access a -tuple/object field operator) and ``[]`` (array/string/sequence index operator) +The empty `[]` subscript notation can be used to *de-refer* a reference, +meaning to retrieve the item the reference points to. The `.` (access a +tuple/object field operator) and `[]` (array/string/sequence index operator) operators perform implicit dereferencing operations for reference types: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" type Node = ref object le, ri: Node data: int - var - n: Node - new(n) - n.data = 9 + + var n = Node(data: 9) + echo n.data # no need to write n[].data; in fact n[].data is highly discouraged! + ``` + +To allocate a new traced object, the built-in procedure `new` can be used: -To allocate a new traced object, the built-in procedure ``new`` must be used. -To deal with untraced memory, the procedures ``alloc``, ``dealloc`` and -``realloc`` can be used. The `system <system.html>`_ + ```nim + var n: Node + new(n) + ``` + +To deal with untraced memory, the procedures `alloc`, `dealloc` and +`realloc` can be used. The [system](system.html) module's documentation contains further details. -If a reference points to *nothing*, it has the value ``nil``. +If a reference points to *nothing*, it has the value `nil`. Procedural type --------------- + A procedural type is a (somewhat abstract) pointer to a procedure. -``nil`` is an allowed value for a variable of a procedural type. +`nil` is an allowed value for a variable of a procedural type. Nim uses procedural types to achieve `functional`:idx: programming techniques. Example: -.. code-block:: nim - :test: "nim c $1" - proc echoItem(x: int) = echo x + ```nim test = "nim c $1" + proc greet(name: string): string = + "Hello, " & name & "!" + + proc bye(name: string): string = + "Goodbye, " & name & "." - proc forEach(action: proc (x: int)) = - const - data = [2, 3, 5, 7, 11] - for d in items(data): - action(d) + proc communicate(greeting: proc (x: string): string, name: string) = + echo greeting(name) - forEach(echoItem) + communicate(greet, "John") + communicate(bye, "Mary") + ``` A subtle issue with procedural types is that the calling convention of the procedure influences the type compatibility: procedural types are only compatible if they have the same calling convention. The different calling conventions are -listed in the `manual <manual.html#types-procedural-type>`_. +listed in the [manual](manual.html#types-procedural-type). Distinct type ------------- -A Distinct type allows for the creation of new type that "does not imply a + +A Distinct type allows for the creation of a new type that "does not imply a subtype relationship between it and its base type". -You must **explicitly** define all behaviour for the distinct type. +You must **explicitly** define all behavior for the distinct type. To help with this, both the distinct type and its base type can cast from one type to the other. -Examples are provided in the `manual <manual.html#types-distinct-type>`_. +Examples are provided in the [manual](manual.html#types-distinct-type). Modules ======= -Nim supports splitting a program into pieces with a module concept. + +Nim supports splitting a program into pieces with a *module* concept. Each module is in its own file. Modules enable `information hiding`:idx: and `separate compilation`:idx:. A module may gain access to the symbols of another module by using the `import`:idx: statement. Only top-level symbols that are marked -with an asterisk (``*``) are exported: +with an asterisk (`*`) are exported: -.. code-block:: nim + ```nim # Module A var x*, y: int @@ -1702,35 +1811,38 @@ with an asterisk (``*``) are exported: # allocate a new sequence: newSeq(result, len(a)) # multiply two int sequences: - for i in 0..len(a)-1: result[i] = a[i] * b[i] + for i in 0 ..< len(a): result[i] = a[i] * b[i] when isMainModule: - # test the new ``*`` operator for sequences: + # test the new `*` operator for sequences: assert(@[1, 2, 3] * @[1, 2, 3] == @[1, 4, 9]) + ``` -The above module exports ``x`` and ``*``, but not ``y``. +The above module exports `x` and `*`, but not `y`. A module's top-level statements are executed at the start of the program. This can be used to initialize complex data structures for example. -Each module has a special magic constant ``isMainModule`` that is true if the +Each module has a special magic constant `isMainModule` that is true if the module is compiled as the main file. This is very useful to embed tests within the module as shown by the above example. -A symbol of a module *can* be *qualified* with the ``module.symbol`` syntax. And if +A symbol of a module *can* be *qualified* with the `module.symbol` syntax. And if a symbol is ambiguous, it *must* be qualified. A symbol is ambiguous if it is defined in two (or more) different modules and both modules are imported by a third one: -.. code-block:: nim + ```nim # Module A var x*: string + ``` -.. code-block:: nim + ```nim # Module B var x*: int + ``` -.. code-block:: nim + ```nim # Module C import A, B write(stdout, x) # error: x is ambiguous @@ -1738,20 +1850,23 @@ imported by a third one: var x = 4 write(stdout, x) # not ambiguous: uses the module C's x + ``` But this rule does not apply to procedures or iterators. Here the overloading rules apply: -.. code-block:: nim + ```nim # Module A proc x*(a: int): string = $a + ``` -.. code-block:: nim + ```nim # Module B proc x*(a: string): string = $a + ``` -.. code-block:: nim + ```nim # Module C import A, B write(stdout, x(3)) # no error: A.x is called @@ -1759,62 +1874,70 @@ rules apply: proc x*(a: int): string = discard write(stdout, x(3)) # ambiguous: which `x` is to call? + ``` Excluding symbols ----------------- -The normal ``import`` statement will bring in all exported symbols. -These can be limited by naming symbols which should be excluded with -the ``except`` qualifier. +The normal `import` statement will bring in all exported symbols. +These can be limited by naming symbols that should be excluded using +the `except` qualifier. -.. code-block:: nim + ```nim import mymodule except y + ``` From statement -------------- -We have already seen the simple ``import`` statement that just imports all +We have already seen the simple `import` statement that just imports all exported symbols. An alternative that only imports listed symbols is the -``from import`` statement: +`from import` statement: -.. code-block:: nim + ```nim from mymodule import x, y, z + ``` -The ``from`` statement can also force namespace qualification on +The `from` statement can also force namespace qualification on symbols, thereby making symbols available, but needing to be qualified -to be used. +in order to be used. -.. code-block:: nim + ```nim from mymodule import x, y, z x() # use x without any qualification + ``` -.. code-block:: nim + ```nim from mymodule import nil mymodule.x() # must qualify x with the module name as prefix x() # using x here without qualification is a compile error + ``` Since module names are generally long to be descriptive, you can also define a shorter alias to use when qualifying symbols. -.. code-block:: nim + ```nim from mymodule as m import nil m.x() # m is aliasing mymodule + ``` Include statement ----------------- -The ``include`` statement does something fundamentally different than -importing a module: it merely includes the contents of a file. The ``include`` + +The `include` statement does something fundamentally different than +importing a module: it merely includes the contents of a file. The `include` statement is useful to split up a large module into several files: -.. code-block:: nim + ```nim include fileA, fileB, fileC + ``` @@ -1822,7 +1945,7 @@ Part 2 ====== So, now that we are done with the basics, let's see what Nim offers apart -from a nice syntax for procedural programming: `Part II <tut2.html>`_ +from a nice syntax for procedural programming: [Part II](tut2.html) .. _strutils: strutils.html diff --git a/doc/tut2.rst b/doc/tut2.md index ac8e82b0a..1b59288d5 100644 --- a/doc/tut2.rst +++ b/doc/tut2.md @@ -5,17 +5,19 @@ Nim Tutorial (Part II) :Author: Andreas Rumpf :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: Introduction ============ - "Repetition renders the ridiculous reasonable." -- Norman Wildberger +> "Repetition renders the ridiculous reasonable." -- Norman Wildberger This document is a tutorial for the advanced constructs of the *Nim* programming language. **Note that this document is somewhat obsolete as the** -`manual <manual.html>`_ **contains many more examples of the advanced language +[manual](manual.html) **contains many more examples of the advanced language features.** @@ -24,9 +26,9 @@ Pragmas Pragmas are Nim's method to give the compiler additional information/ commands without introducing a massive number of new keywords. Pragmas are -enclosed in the special ``{.`` and ``.}`` curly dot brackets. This tutorial -does not cover pragmas. See the `manual <manual.html#pragmas>`_ or `user guide -<nimc.html#additional-features>`_ for a description of the available +enclosed in the special `{.` and `.}` curly dot brackets. This tutorial +does not cover pragmas. See the [manual](manual.html#pragmas) or [user guide]( +nimc.html#additional-features) for a description of the available pragmas. @@ -45,14 +47,13 @@ Inheritance Inheritance in Nim is entirely optional. To enable inheritance with runtime type information the object needs to inherit from -``RootObj``. This can be done directly, or indirectly by -inheriting from an object that inherits from ``RootObj``. Usually -types with inheritance are also marked as ``ref`` types even though +`RootObj`. This can be done directly, or indirectly by +inheriting from an object that inherits from `RootObj`. Usually +types with inheritance are also marked as `ref` types even though this isn't strictly enforced. To check at runtime if an object is of a certain -type, the ``of`` operator can be used. +type, the `of` operator can be used. -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type Person = ref object of RootObj name*: string # the * means that `name` is accessible from other modules @@ -68,17 +69,18 @@ type, the ``of`` operator can be used. # object construction: student = Student(name: "Anton", age: 5, id: 2) echo student[] + ``` -Inheritance is done with the ``object of`` syntax. Multiple inheritance is -currently not supported. If an object type has no suitable ancestor, ``RootObj`` +Inheritance is done with the `object of` syntax. Multiple inheritance is +currently not supported. If an object type has no suitable ancestor, `RootObj` can be used as its ancestor, but this is only a convention. Objects that have -no ancestor are implicitly ``final``. You can use the ``inheritable`` pragma -to introduce new object roots apart from ``system.RootObj``. (This is used +no ancestor are implicitly `final`. You can use the `inheritable` pragma +to introduce new object roots apart from `system.RootObj`. (This is used in the GTK wrapper for instance.) Ref objects should be used whenever inheritance is used. It isn't strictly -necessary, but with non-ref objects assignments such as ``let person: Person = -Student(id: 123)`` will truncate subclass fields. +necessary, but with non-ref objects, assignments such as `let person: Person = +Student(id: 123)` will truncate subclass fields. **Note**: Composition (*has-a* relation) is often preferable to inheritance (*is-a* relation) for simple code reuse. Since objects are value types in @@ -95,8 +97,7 @@ would require arbitrary symbol lookahead which slows down compilation.) Example: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type Node = ref object # a reference to an object with the following field: le, ri: Node # left and right subtrees @@ -106,12 +107,13 @@ Example: name: string # the symbol's name line: int # the line the symbol was declared in code: Node # the symbol's abstract syntax tree + ``` Type conversions ---------------- Nim distinguishes between `type casts`:idx: and `type conversions`:idx:. -Casts are done with the ``cast`` operator and force the compiler to +Casts are done with the `cast` operator and force the compiler to interpret a bit pattern to be of another type. Type conversions are a much more polite way to convert a type into another: @@ -119,15 +121,16 @@ They preserve the abstract *value*, not necessarily the *bit-pattern*. If a type conversion is not possible, the compiler complains or an exception is raised. -The syntax for type conversions is ``destination_type(expression_to_convert)`` +The syntax for type conversions is `destination_type(expression_to_convert)` (like an ordinary call): -.. code-block:: nim + ```nim proc getID(x: Person): int = Student(x).id + ``` -The ``InvalidObjectConversionDefect`` exception is raised if ``x`` is not a -``Student``. +The `InvalidObjectConversionDefect` exception is raised if `x` is not a +`Student`. Object variants @@ -137,9 +140,7 @@ variant types are needed. An example: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" # This is an example how an abstract syntax tree could be modelled in Nim type NodeKind = enum # the different node types @@ -150,7 +151,7 @@ An example: nkSub, # a subtraction nkIf # an if statement Node = ref object - case kind: NodeKind # the ``kind`` field is the discriminator + case kind: NodeKind # the `kind` field is the discriminator of nkInt: intVal: int of nkFloat: floatVal: float of nkString: strVal: string @@ -163,8 +164,9 @@ An example: # the following statement raises an `FieldDefect` exception, because # n.kind's value does not fit: n.strVal = "" + ``` -As can been seen from the example, an advantage to an object hierarchy is that +As can be seen from the example, an advantage to an object hierarchy is that no conversion between different object types is needed. Yet, access to invalid object fields raises an exception. @@ -173,34 +175,35 @@ Method call syntax ------------------ There is a syntactic sugar for calling routines: -The syntax ``obj.method(args)`` can be used instead of ``method(obj, args)``. +The syntax `obj.methodName(args)` can be used +instead of `methodName(obj, args)`. If there are no remaining arguments, the parentheses can be omitted: -``obj.len`` (instead of ``len(obj)``). +`obj.len` (instead of `len(obj)`). This method call syntax is not restricted to objects, it can be used for any type: -.. code-block:: nim - :test: "nim c $1" - import strutils + ```nim test = "nim c $1" + import std/strutils echo "abc".len # is the same as echo len("abc") echo "abc".toUpperAscii() echo({'a', 'b', 'c'}.card) stdout.writeLine("Hallo") # the same as writeLine(stdout, "Hallo") + ``` (Another way to look at the method call syntax is that it provides the missing postfix notation.) So "pure object oriented" code is easy to write: -.. code-block:: nim - :test: "nim c $1" - import strutils, sequtils + ```nim test = "nim c $1" + import std/[strutils, sequtils] stdout.writeLine("Give a list of numbers (separated by spaces): ") stdout.write(stdin.readLine.splitWhitespace.map(parseInt).max.`$`) stdout.writeLine(" is the maximum!") + ``` Properties @@ -210,9 +213,7 @@ Ordinary get-procedures that are called with the *method call syntax* achieve the same. But setting a value is different; for this a special setter syntax is needed: -.. code-block:: nim - :test: "nim c $1" - + ```nim test = "nim c $1" type Socket* = ref object of RootObj h: int # cannot be accessed from the outside of the module due to missing star @@ -228,15 +229,15 @@ is needed: var s: Socket new s s.host = 34 # same as `host=`(s, 34) + ``` -(The example also shows ``inline`` procedures.) +(The example also shows `inline` procedures.) -The ``[]`` array access operator can be overloaded to provide +The `[]` array access operator can be overloaded to provide `array properties`:idx:\ : -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type Vector* = object x, y, z: float @@ -256,19 +257,19 @@ The ``[]`` array access operator can be overloaded to provide of 1: result = v.y of 2: result = v.z else: assert(false) + ``` The example is silly, since a vector is better modelled by a tuple which -already provides ``v[]`` access. +already provides `v[]` access. Dynamic dispatch ---------------- Procedures always use static dispatch. For dynamic dispatch replace the -``proc`` keyword by ``method``: +`proc` keyword by `method`: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type Expression = ref object of RootObj ## abstract base class for an expression Literal = ref object of Expression @@ -288,9 +289,10 @@ Procedures always use static dispatch. For dynamic dispatch replace the proc newPlus(a, b: Expression): PlusExpr = PlusExpr(a: a, b: b) echo eval(newPlus(newPlus(newLit(1), newLit(2)), newLit(4))) + ``` -Note that in the example the constructors ``newLit`` and ``newPlus`` are procs -because it makes more sense for them to use static binding, but ``eval`` is a +Note that in the example the constructors `newLit` and `newPlus` are procs +because it makes more sense for them to use static binding, but `eval` is a method because it requires dynamic binding. **Note:** Starting from Nim 0.20, to use multi-methods one must explicitly pass @@ -299,9 +301,7 @@ method because it requires dynamic binding. In a multi-method all parameters that have an object type are used for the dispatching: -.. code-block:: nim - :test: "nim c --multiMethods:on $1" - + ```nim test = "nim c --multiMethods:on $1" type Thing = ref object of RootObj Unit = ref object of Thing @@ -320,11 +320,12 @@ dispatching: new a new b collide(a, b) # output: 2 + ``` As the example demonstrates, invocation of a multi-method cannot be ambiguous: Collide 2 is preferred over collide 1 because the resolution works from left to -right. Thus ``Unit, Thing`` is preferred over ``Thing, Unit``. +right. Thus `Unit, Thing` is preferred over `Thing, Unit`. **Performance note**: Nim does not produce a virtual method table, but generates dispatch trees. This avoids the expensive indirect branch for method @@ -336,48 +337,46 @@ Exceptions ========== In Nim exceptions are objects. By convention, exception types are -suffixed with 'Error'. The `system <system.html>`_ module defines an +suffixed with 'Error'. The [system](system.html) module defines an exception hierarchy that you might want to stick to. Exceptions derive from -``system.Exception``, which provides the common interface. +`system.Exception`, which provides the common interface. Exceptions have to be allocated on the heap because their lifetime is unknown. The compiler will prevent you from raising an exception created on the stack. All raised exceptions should at least specify the reason for being raised in -the ``msg`` field. - -A convention is that exceptions should be raised in *exceptional* cases: -For example, if a file cannot be opened, this should not raise an -exception since this is quite common (the file may not exist). +the `msg` field. +A convention is that exceptions should be raised in *exceptional* cases, +they should not be used as an alternative method of control flow. Raise statement --------------- -Raising an exception is done with the ``raise`` statement: +Raising an exception is done with the `raise` statement: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" var e: ref OSError new(e) e.msg = "the request to the OS failed" raise e + ``` -If the ``raise`` keyword is not followed by an expression, the last exception +If the `raise` keyword is not followed by an expression, the last exception is *re-raised*. For the purpose of avoiding repeating this common code pattern, -the template ``newException`` in the ``system`` module can be used: +the template `newException` in the `system` module can be used: -.. code-block:: nim + ```nim raise newException(OSError, "the request to the OS failed") + ``` Try statement ------------- -The ``try`` statement handles exceptions: +The `try` statement handles exceptions: -.. code-block:: nim - :test: "nim c $1" - from strutils import parseInt + ```nim test = "nim c $1" + from std/strutils import parseInt # read the first two lines of a text file that should contain numbers # and tries to add them @@ -394,60 +393,63 @@ The ``try`` statement handles exceptions: echo "could not convert string to integer" except IOError: echo "IO error!" - except: + except CatchableError: echo "Unknown exception!" # reraise the unknown exception: raise finally: close(f) + ``` -The statements after the ``try`` are executed unless an exception is -raised. Then the appropriate ``except`` part is executed. +The statements after the `try` are executed unless an exception is +raised. Then the appropriate `except` part is executed. -The empty ``except`` part is executed if there is an exception that is -not explicitly listed. It is similar to an ``else`` part in ``if`` +The empty `except` part is executed if there is an exception that is +not explicitly listed. It is similar to an `else` part in `if` statements. -If there is a ``finally`` part, it is always executed after the +If there is a `finally` part, it is always executed after the exception handlers. -The exception is *consumed* in an ``except`` part. If an exception is not +The exception is *consumed* in an `except` part. If an exception is not handled, it is propagated through the call stack. This means that often -the rest of the procedure - that is not within a ``finally`` clause - +the rest of the procedure - that is not within a `finally` clause - is not executed (if an exception occurs). If you need to *access* the actual exception object or message inside an -``except`` branch you can use the `getCurrentException() -<system.html#getCurrentException>`_ and `getCurrentExceptionMsg() -<system.html#getCurrentExceptionMsg>`_ procs from the `system <system.html>`_ +`except` branch you can use the [getCurrentException()]( +system.html#getCurrentException) and [getCurrentExceptionMsg()]( +system.html#getCurrentExceptionMsg) procs from the [system](system.html) module. Example: -.. code-block:: nim + ```nim try: doSomethingHere() - except: + except CatchableError: let e = getCurrentException() msg = getCurrentExceptionMsg() echo "Got exception ", repr(e), " with message ", msg + ``` Annotating procs with raised exceptions --------------------------------------- -Through the use of the optional ``{.raises.}`` pragma you can specify that a +Through the use of the optional `{.raises.}` pragma you can specify that a proc is meant to raise a specific set of exceptions, or none at all. If the -``{.raises.}`` pragma is used, the compiler will verify that this is true. For -instance, if you specify that a proc raises ``IOError``, and at some point it +`{.raises.}` pragma is used, the compiler will verify that this is true. For +instance, if you specify that a proc raises `IOError`, and at some point it (or one of the procs it calls) starts raising a new exception the compiler will prevent that proc from compiling. Usage example: -.. code-block:: nim + ```nim proc complexProc() {.raises: [IOError, ArithmeticDefect].} = ... proc simpleProc() {.raises: [].} = ... + ``` Once you have code like this in place, if the list of raised exception changes the compiler will stop with an error specifying the line of the proc which @@ -455,14 +457,14 @@ stopped validating the pragma and the raised exception not being caught, along with the file and line where the uncaught exception is being raised, which may help you locate the offending code which has changed. -If you want to add the ``{.raises.}`` pragma to existing code, the compiler can -also help you. You can add the ``{.effects.}`` pragma statement to your proc and +If you want to add the `{.raises.}` pragma to existing code, the compiler can +also help you. You can add the `{.effects.}` pragma statement to your proc and the compiler will output all inferred effects up to that point (exception tracking is part of Nim's effect system). Another more roundabout way to -find out the list of exceptions raised by a proc is to use the Nim ``doc2`` +find out the list of exceptions raised by a proc is to use the Nim ``doc`` command which generates documentation for a whole module and decorates all procs with the list of raised exceptions. You can read more about Nim's -`effect system and related pragmas in the manual <manual.html#effect-system>`_. +[effect system and related pragmas in the manual](manual.html#effect-system). Generics @@ -470,14 +472,13 @@ Generics Generics are Nim's means to parametrize procs, iterators or types with `type parameters`:idx:. Generic parameters are written within square -brackets, for example ``Foo[T]``. They are most useful for efficient type safe +brackets, for example `Foo[T]`. They are most useful for efficient type safe containers: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" type BinaryTree*[T] = ref object # BinaryTree is a generic type with - # generic param ``T`` + # generic param `T` le, ri: BinaryTree[T] # left and right subtrees; may be nil data: T # the data stored in a node @@ -493,8 +494,8 @@ containers: else: var it = root while it != nil: - # compare the data items; uses the generic ``cmp`` proc - # that works for any type that has a ``==`` and ``<`` operator + # compare the data items; uses the generic `cmp` proc + # that works for any type that has a `==` and `<` operator var c = cmp(it.data, n.data) if c < 0: if it.le == nil: @@ -513,8 +514,8 @@ containers: iterator preorder*[T](root: BinaryTree[T]): T = # Preorder traversal of a binary tree. - # Since recursive iterators are not yet implemented, - # this uses an explicit stack (which is more efficient anyway): + # This uses an explicit stack (which is more efficient than + # a recursive iterator factory). var stack: seq[BinaryTree[T]] = @[root] while stack.len > 0: var n = stack.pop() @@ -524,17 +525,31 @@ containers: n = n.le # and follow the left pointer var - root: BinaryTree[string] # instantiate a BinaryTree with ``string`` - add(root, newNode("hello")) # instantiates ``newNode`` and ``add`` - add(root, "world") # instantiates the second ``add`` proc + root: BinaryTree[string] # instantiate a BinaryTree with `string` + add(root, newNode("hello")) # instantiates `newNode` and `add` + add(root, "world") # instantiates the second `add` proc for str in preorder(root): stdout.writeLine(str) + ``` The example shows a generic binary tree. Depending on context, the brackets are used either to introduce type parameters or to instantiate a generic proc, iterator or type. As the example shows, generics work with overloading: the -best match of ``add`` is used. The built-in ``add`` procedure for sequences -is not hidden and is used in the ``preorder`` iterator. +best match of `add` is used. The built-in `add` procedure for sequences +is not hidden and is used in the `preorder` iterator. + +There is a special `[:T]` syntax when using generics with the method call syntax: + + ```nim test = "nim c $1" + proc foo[T](i: T) = + discard + + var i: int + + # i.foo[int]() # Error: expression 'foo(i)' has no type (or is ambiguous) + + i.foo[:int]() # Success + ``` Templates @@ -549,27 +564,27 @@ To *invoke* a template, call it like a procedure. Example: -.. code-block:: nim + ```nim template `!=` (a, b: untyped): untyped = # this definition exists in the System module not (a == b) assert(5 != 6) # the compiler rewrites that to: assert(not (5 == 6)) + ``` -The ``!=``, ``>``, ``>=``, ``in``, ``notin``, ``isnot`` operators are in fact -templates: this has the benefit that if you overload the ``==`` operator, -the ``!=`` operator is available automatically and does the right thing. (Except +The `!=`, `>`, `>=`, `in`, `notin`, `isnot` operators are in fact +templates: this has the benefit that if you overload the `==` operator, +the `!=` operator is available automatically and does the right thing. (Except for IEEE floating point numbers - NaN breaks basic boolean logic.) -``a > b`` is transformed into ``b < a``. -``a in b`` is transformed into ``contains(b, a)``. -``notin`` and ``isnot`` have the obvious meanings. +`a > b` is transformed into `b < a`. +`a in b` is transformed into `contains(b, a)`. +`notin` and `isnot` have the obvious meanings. Templates are especially useful for lazy evaluation purposes. Consider a simple proc for logging: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" const debug = true @@ -579,15 +594,15 @@ simple proc for logging: var x = 4 log("x has the value: " & $x) + ``` -This code has a shortcoming: if ``debug`` is set to false someday, the quite -expensive ``$`` and ``&`` operations are still performed! (The argument +This code has a shortcoming: if `debug` is set to false someday, the quite +expensive `$` and `&` operations are still performed! (The argument evaluation for procedures is *eager*). -Turning the ``log`` proc into a template solves this problem: +Turning the `log` proc into a template solves this problem: -.. code-block:: nim - :test: "nim c $1" + ```nim test = "nim c $1" const debug = true @@ -597,20 +612,19 @@ Turning the ``log`` proc into a template solves this problem: var x = 4 log("x has the value: " & $x) + ``` -The parameters' types can be ordinary types or the meta types ``untyped``, -``typed``, or ``type``. ``type`` suggests that only a type symbol may be given -as an argument, and ``untyped`` means symbol lookups and type resolution is not +The parameters' types can be ordinary types or the meta types `untyped`, +`typed`, or `type`. `type` suggests that only a type symbol may be given +as an argument, and `untyped` means symbol lookups and type resolution is not performed before the expression is passed to the template. If the template has no explicit return type, -``void`` is used for consistency with procs and methods. - -To pass a block of statements to a template, use ``untyped`` for the last parameter: +`void` is used for consistency with procs and methods. -.. code-block:: nim - :test: "nim c $1" +To pass a block of statements to a template, use `untyped` for the last parameter: + ```nim test = "nim c $1" template withFile(f: untyped, filename: string, mode: FileMode, body: untyped) = let fn = filename @@ -626,29 +640,30 @@ To pass a block of statements to a template, use ``untyped`` for the last parame withFile(txt, "ttempl3.txt", fmWrite): txt.writeLine("line 1") txt.writeLine("line 2") + ``` -In the example the two ``writeLine`` statements are bound to the ``body`` -parameter. The ``withFile`` template contains boilerplate code and helps to +In the example the two `writeLine` statements are bound to the `body` +parameter. The `withFile` template contains boilerplate code and helps to avoid a common bug: to forget to close the file. Note how the -``let fn = filename`` statement ensures that ``filename`` is evaluated only +`let fn = filename` statement ensures that `filename` is evaluated only once. Example: Lifting Procs ---------------------- -.. code-block:: nim - :test: "nim c $1" - import math + `````nim test = "nim c $1" + import std/math template liftScalarProc(fname) = ## Lift a proc taking one scalar parameter and returning a - ## scalar value (eg ``proc sssss[T](x: T): float``), + ## scalar value (eg `proc sssss[T](x: T): float`), ## to provide templated procs that can handle a single ## parameter of seq[T] or nested seq[seq[]] or the same type ## - ## .. code-block:: Nim - ## liftScalarProc(abs) - ## # now abs(@[@[1,-2], @[-2,-3]]) == @[@[1,2], @[2,3]] + ## ```Nim + ## liftScalarProc(abs) + ## # now abs(@[@[1,-2], @[-2,-3]]) == @[@[1,2], @[2,3]] + ## ``` proc fname[T](x: openarray[T]): auto = var temp: T type outType = typeof(fname(temp)) @@ -658,24 +673,25 @@ Example: Lifting Procs liftScalarProc(sqrt) # make sqrt() work for sequences echo sqrt(@[4.0, 16.0, 25.0, 36.0]) # => @[2.0, 4.0, 5.0, 6.0] + ````` Compilation to JavaScript ========================= Nim code can be compiled to JavaScript. However in order to write JavaScript-compatible code you should remember the following: -- ``addr`` and ``ptr`` have slightly different semantic meaning in JavaScript. +- `addr` and `ptr` have slightly different semantic meaning in JavaScript. It is recommended to avoid those if you're not sure how they are translated to JavaScript. -- ``cast[T](x)`` in JavaScript is translated to ``(x)``, except for casting +- `cast[T](x)` in JavaScript is translated to `(x)`, except for casting between signed/unsigned ints, in which case it behaves as static cast in C language. -- ``cstring`` in JavaScript means JavaScript string. It is a good practice to - use ``cstring`` only when it is semantically appropriate. E.g. don't use - ``cstring`` as a binary data buffer. +- `cstring` in JavaScript means JavaScript string. It is a good practice to + use `cstring` only when it is semantically appropriate. E.g. don't use + `cstring` as a binary data buffer. Part 3 ====== -The next part is entirely about metaprogramming via macros: `Part III <tut3.html>`_ +The next part is entirely about metaprogramming via macros: [Part III](tut3.html). diff --git a/doc/tut3.rst b/doc/tut3.md index a39074db9..3a55d4790 100644 --- a/doc/tut3.rst +++ b/doc/tut3.md @@ -5,45 +5,47 @@ Nim Tutorial (Part III) :Author: Arne Döring :Version: |nimversion| +.. default-role:: code +.. include:: rstcommon.rst .. contents:: Introduction ============ - "With Great Power Comes Great Responsibility." -- Spider Man's Uncle +> "With Great Power Comes Great Responsibility." -- Spider Man's Uncle This document is a tutorial about Nim's macro system. -A macro is a function that is executed at compile time and transforms +A macro is a function that is executed at compile-time and transforms a Nim syntax tree into a different tree. Examples of things that can be implemented in macros: * An assert macro that prints both sides of a comparison operator, if - the assertion fails. ``myAssert(a == b)`` is converted to - ``if a != b: quit($a " != " $b)`` + the assertion fails. `myAssert(a == b)` is converted to + `if a != b: quit($a " != " $b)` * A debug macro that prints the value and the name of the symbol. - ``myDebugEcho(a)`` is converted to ``echo "a: ", a`` + `myDebugEcho(a)` is converted to `echo "a: ", a` * Symbolic differentiation of an expression. - ``diff(a*pow(x,3) + b*pow(x,2) + c*x + d, x)`` is converted to - ``3*a*pow(x,2) + 2*b*x + c`` + `diff(a*pow(x,3) + b*pow(x,2) + c*x + d, x)` is converted to + `3*a*pow(x,2) + 2*b*x + c` Macro Arguments --------------- The types of macro arguments have two faces. One face is used for -the overload resolution, and the other face is used within the macro -body. For example, if ``macro foo(arg: int)`` is called in an -expression ``foo(x)``, ``x`` has to be of a type compatible to int, but -*within* the macro's body ``arg`` has the type ``NimNode``, not ``int``! +the overload resolution and the other face is used within the macro +body. For example, if `macro foo(arg: int)` is called in an +expression `foo(x)`, `x` has to be of a type compatible to int, but +*within* the macro's body `arg` has the type `NimNode`, not `int`! Why it is done this way will become obvious later, when we have seen concrete examples. There are two ways to pass arguments to a macro, an argument can be -either ``typed`` or ``untyped``. +either `typed` or `untyped`. Untyped Arguments @@ -52,17 +54,17 @@ Untyped Arguments Untyped macro arguments are passed to the macro before they are semantically checked. This means the syntax tree that is passed down to the macro does not need to make sense for Nim yet, the only -limitation is that it needs to be parsable. Usually the macro does +limitation is that it needs to be parsable. Usually, the macro does not check the argument either but uses it in the transformation's result somehow. The result of a macro expansion is always checked -by the compiler, so apart from weird error messages nothing bad +by the compiler, so apart from weird error messages, nothing bad can happen. -The downside for an ``untyped`` argument is that these do not play +The downside for an `untyped` argument is that these do not play well with Nim's overloading resolution. The upside for untyped arguments is that the syntax tree is -quite predictable and less complex compared to its ``typed`` +quite predictable and less complex compared to its `typed` counterpart. @@ -73,9 +75,9 @@ For typed arguments, the semantic checker runs on the argument and does transformations on it, before it is passed to the macro. Here identifier nodes are resolved as symbols, implicit type conversions are visible in the tree as calls, templates are -expanded and probably most importantly, nodes have type information. -Typed arguments can have the type ``typed`` in the arguments list. -But all other types, such as ``int``, ``float`` or ``MyObjectType`` +expanded, and probably most importantly, nodes have type information. +Typed arguments can have the type `typed` in the arguments list. +But all other types, such as `int`, `float` or `MyObjectType` are typed arguments as well, and they are passed to the macro as a syntax tree. @@ -84,34 +86,34 @@ Static Arguments ---------------- Static arguments are a way to pass values as values and not as syntax -tree nodes to a macro. For example for ``macro foo(arg: static[int])`` -in the expression ``foo(x)``, ``x`` needs to be an integer constant, -but in the macro body ``arg`` is just like a normal parameter of type -``int``. +tree nodes to a macro. For example for `macro foo(arg: static[int])` +in the expression `foo(x)`, `x` needs to be an integer constant, +but in the macro body `arg` is just like a normal parameter of type +`int`. -.. code-block:: nim - - import macros + ```nim + import std/macros macro myMacro(arg: static[int]): untyped = - echo arg # just an int (7), not ``NimNode`` + echo arg # just an int (7), not `NimNode` myMacro(1 + 2 * 3) + ``` Code Blocks as Arguments ------------------------ It is possible to pass the last argument of a call expression in a -separate code block with indentation. For example the following code -example is a valid (but not a recommended) way to call ``echo``: - -.. code-block:: nim +separate code block with indentation. For example, the following code +example is a valid (but not a recommended) way to call `echo`: + ```nim echo "Hello ": let a = "Wor" let b = "ld!" a & b + ``` For macros this way of calling is very useful; syntax trees of arbitrary complexity can be passed to macros with this notation. @@ -123,17 +125,16 @@ The Syntax Tree In order to build a Nim syntax tree one needs to know how Nim source code is represented as a syntax tree, and how such a tree needs to look like so that the Nim compiler will understand it. The nodes of the -Nim syntax tree are documented in the `macros <macros.html>`_ module. +Nim syntax tree are documented in the [macros](macros.html) module. But a more interactive way to explore the Nim -syntax tree is with ``macros.treeRepr``, it converts a syntax tree -into a multi line string for printing on the console. It can be used +syntax tree is with `macros.treeRepr`, it converts a syntax tree +into a multi-line string for printing on the console. It can be used to explore how the argument expressions are represented in tree form -and for debug printing of generated syntax tree. ``dumpTree`` is a -predefined macro that just prints its argument in tree representation, +and for debug printing of generated syntax tree. `dumpTree` is a +predefined macro that just prints its argument in a tree representation, but does nothing else. Here is an example of such a tree representation: -.. code-block:: nim - + ```nim dumpTree: var mt: MyType = MyType(a:123.456, b:"abcdef") @@ -151,49 +152,80 @@ but does nothing else. Here is an example of such a tree representation: # ExprColonExpr # Ident "b" # StrLit "abcdef" + ``` Custom Semantic Checking ------------------------ +------------------------ The first thing that a macro should do with its arguments is to check if the argument is in the correct form. Not every type of wrong input needs to be caught here, but anything that could cause a crash during macro evaluation should be caught and create a nice error message. -``macros.expectKind`` and ``macros.expectLen`` are a good start. If +`macros.expectKind` and `macros.expectLen` are a good start. If the checks need to be more complex, arbitrary error messages can -be created with the ``macros.error`` proc. - -.. code-block:: nim +be created with the `macros.error` proc. + ```nim macro myAssert(arg: untyped): untyped = arg.expectKind nnkInfix + ``` Generating Code --------------- There are two ways to generate the code. Either by creating the syntax -tree with expressions that contain a lot of calls to ``newTree`` and -``newLit``, or with ``quote do:`` expressions. The first option offers -the best low level control for the syntax tree generation, but the +tree with expressions that contain a lot of calls to `newTree` and +`newLit`, or with `quote do:` expressions. The first option offers +the best low-level control for the syntax tree generation, but the second option is much less verbose. If you choose to create the syntax -tree with calls to ``newTree`` and ``newLit`` the macro -``macros.dumpAstGen`` can help you with the verbosity. ``quote do:`` -allows you to write the code that you want to generate literally, -backticks are used to insert code from ``NimNode`` symbols into the -generated expression. This means that you can't use backticks within -``quote do:`` for anything else than injecting symbols. Make sure to -inject only symbols of type ``NimNode`` into the generated syntax -tree. You can use ``newLit`` to convert arbitrary values into -expressions trees of type ``NimNode`` so that it is safe to inject -them into the tree. +tree with calls to `newTree` and `newLit` the macro +`macros.dumpAstGen` can help you with the verbosity. + +`quote do:` allows you to write the code that you want to generate literally. +Backticks are used to insert code from `NimNode` symbols into the +generated expression. + + ```nim test = "nim c $1" + import std/macros + macro a(i) = quote do: + let `i` = 0 + a b + doAssert b == 0 + ``` -.. code-block:: nim - :test: "nim c $1" +A custom prefix operator can be defined whenever backticks are needed. + + ```nim test = "nim c $1" + import std/macros + macro a(i) = quote("@") do: + assert @i == 0 + + let b = 0 + a b + ``` + +The injected symbol needs accent quoted when it resolves to a symbol. + + ```nim test = "nim c $1" + import std/macros + macro a(i) = quote("@") do: + let `@i` = 0 + + a b + doAssert b == 0 + ``` + +Make sure to inject only symbols of type `NimNode` into the generated syntax +tree. You can use `newLit` to convert arbitrary values into +expressions trees of type `NimNode` so that it is safe to inject +them into the tree. - import macros + + ```nim test = "nim c $1" + import std/macros type MyType = object @@ -212,27 +244,27 @@ them into the tree. echo `mtLit` myMacro("Hallo") + ``` -The call to ``myMacro`` will generate the following code: +The call to `myMacro` will generate the following code: -.. code-block:: nim + ```nim echo "Hallo" echo MyType(a: 123.456'f64, b: "abcdef") + ``` Building Your First Macro ------------------------- To give a starting point to writing macros we will show now how to -implement the ``myDebug`` macro mentioned earlier. The first thing to +implement the `myAssert` macro mentioned earlier. The first thing to do is to build a simple example of the macro usage, and then just -print the argument. This way it is possible to get an idea of a -correct argument should be look like. - -.. code-block:: nim - :test: "nim c $1" +print the argument. This way it is possible to get an idea of what a +correct argument should look like. - import macros + ```nim test = "nim c $1" + import std/macros macro myAssert(arg: untyped): untyped = echo arg.treeRepr @@ -241,24 +273,23 @@ correct argument should be look like. let b = 2 myAssert(a != b) + ``` -.. code-block:: - + ``` Infix Ident "!=" Ident "a" Ident "b" + ``` -From the output it is possible to see that the argument is an infix +From the output, it is possible to see that the argument is an infix operator (node kind is "Infix"), as well as that the two operands are -at index 1 and 2. With this information the actual macro can be +at index 1 and 2. With this information, the actual macro can be written. -.. code-block:: nim - :test: "nim c $1" - - import macros + ```nim test = "nim c $1" + import std/macros macro myAssert(arg: untyped): untyped = # all node kind identifiers are prefixed with "nnk" @@ -278,27 +309,59 @@ written. myAssert(a != b) myAssert(a == b) + ``` This is the code that will be generated. To debug what the macro -actually generated, the statement ``echo result.repr`` can be used, in +actually generated, the statement `echo result.repr` can be used, in the last line of the macro. It is also the statement that has been used to get this output. -.. code-block:: nim + ```nim if not (a != b): raise newException(AssertionDefect, $a & " != " & $b) + ``` + + +Going further +------------- + +It is possible to create more complex macros by combining different +`NimNode` symbols with `quote do:` expressions. For example, you may +use `newStmtList` to build your macro iteratively, and `ident` in cases +in which you wish to create an identifier from a string, as shown below. + + ```nim + import std/macros + + macro createProcedures() = + result = newStmtList() + + for i in 0..<10: + let name = ident("myProc" & $i) + let content = newLit("I am procedure number #" & $i) + + result.add quote do: + proc `name`() = + echo `content` + + createProcedures() + myProc7() + ``` + +The call to `myProc7` will echo `I am procedure number #7`. + With Power Comes Responsibility ------------------------------- -Macros are very powerful. A good advice is to use them as little as +Macros are very powerful. A piece of good advice is to use them as little as possible, but as much as necessary. Macros can change the semantics of expressions, making the code incomprehensible for anybody who does not know exactly what the macro does with it. So whenever a macro is not necessary and the same logic can be implemented using templates or generics, it is probably better not to use a macro. And when a macro -is used for something, the macro should better have a well written +is used for something, the macro should better have a well-written documentation. For all the people who claim to write only perfectly self-explanatory code: when it comes to macros, the implementation is not enough for documentation. @@ -309,7 +372,7 @@ Limitations Since macros are evaluated in the compiler in the NimVM, macros share all the limitations of the NimVM. They have to be implemented in pure Nim code. Macros can start external processes on the shell, but they -cannot call C functions except from those that are built in the +cannot call C functions except those that are built in the compiler. @@ -324,16 +387,16 @@ possible with it. Strformat --------- -In the Nim standard library, the ``strformat`` library provides a +In the Nim standard library, the `strformat` library provides a macro that parses a string literal at compile time. Parsing a string in a macro like here is generally not recommended. The parsed AST cannot have type information, and parsing implemented on the VM is generally not very fast. Working on AST nodes is almost always the -recommended way. But still ``strformat`` is a good example for a +recommended way. But still `strformat` is a good example for a practical use case for a macro that is slightly more complex than the -``assert`` macro. +`assert` macro. -`Strformat <https://github.com/nim-lang/Nim/blob/5845716df8c96157a047c2bd6bcdd795a7a2b9b1/lib/pure/strformat.nim#L280>`_ +[Strformat](https://github.com/nim-lang/Nim/blob/devel/lib/pure/strformat.nim) Ast Pattern Matching -------------------- @@ -342,13 +405,13 @@ Ast Pattern Matching is a macro library to aid in writing complex macros. This can be seen as a good example of how to repurpose the Nim syntax tree with new semantics. -`Ast Pattern Matching <https://github.com/krux02/ast-pattern-matching>`_ +[Ast Pattern Matching](https://github.com/nim-lang/ast-pattern-matching) OpenGL Sandbox -------------- This project has a working Nim to GLSL compiler written entirely in -macros. It scans recursively though all used function symbols to +macros. It scans recursively through all used function symbols to compile them so that cross library functions can be executed on the GPU. -`OpenGL Sandbox <https://github.com/krux02/opengl-sandbox>`_ +[OpenGL Sandbox](https://github.com/krux02/opengl-sandbox) |