Update nim 2.0 documentation for docgen & nimgrep (#21665)

* Update nim 2.0 documentation for docgen & nimgrep

* [skip ci] Update changelogs/changelog_2_0_0.md

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

* [skip ci] expand section on Markdown migration and `doctype`

---------

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

3 files changed, 102 insertions, 31 deletions

diff --git a/changelogs/changelog_2_0_0.md b/changelogs/changelog_2_0_0.md
index 02310eea3..89759d34e 100644
--- a/changelogs/changelog_2_0_0.md
+++ b/changelogs/changelog_2_0_0.md
@@ -442,7 +442,50 @@
 
 - When compiling for Release the flag `-fno-math-errno` is used for GCC.
 
+## Docgen
+
+- `Markdown` is now default markup language of doc comments (instead
+  of legacy `RstMarkdown` mode). In this release we begin to separate
+  RST and Markdown features to better follow specification of each
+  language, with the focus on Markdown development.
+
+  * So we add `{.doctype: Markdown | RST | RstMarkdown.}` pragma allowing to
+    select the markup language mode in the doc comments of current `.nim`
+    file for processing by `nim doc`:
+
+      1. `Markdown` (default) is basically CommonMark (standard Markdown) +
+         some Pandoc Markdown features + some RST features that are missing
+         in our current implementation of CommonMark and Pandoc Markdown.
+      2. `RST` closely follows RST spec with few additional Nim features.
+      3. `RstMarkdown` is a maximum mix of RST and Markdown features, which
+          is kept for the sake of compatibility and ease of migration.
+
+  * and we add separate `md2html` and `rst2html` commands for processing
+    standalone `.md` and `.rst` files respectively (and also `md2tex/rst2tex`).
+
+- Added Pandoc Markdown bracket syntax `[...]` for making anchor-less links.
+- Docgen now supports concise syntax for referencing Nim symbols:
+  instead of specifying HTML anchors directly one can use original
+  Nim symbol declarations (adding the aforementioned link brackets
+  `[...]` around them).
+    * to use this feature across modules a new `importdoc` directive is added.
+  Using this feature for referencing also helps to ensure that links
+  (inside one module or the whole project) are not broken.
+- Added support for RST & Markdown quote blocks (blocks starting from `>`).
+- Added a popular Markdown definition lists extension.
+- Added Markdown indented code blocks (blocks indented by >= 4 spaces).
+- Added syntax for additional parameters to Markdown code blocks:
+
+       ```nim test="nim c $1"
+       ...
+       ```
 
 ## Tool changes
 
 - Nim now ships Nimble version 0.14 which added support for lock-files. Libraries are stored in `$nimbleDir/pkgs2` (it was `$nimbleDir/pkgs`). Use `nimble develop --global` to create an old style link file in the special links directory documented at https://github.com/nim-lang/nimble#nimble-develop.
+- nimgrep added option `--inContext` (and `--notInContext`), which
+  allows to filter only matches with context block containing a given pattern.
+- nimgrep: names of options containing "include/exclude" are deprecated,
+  e.g. instead of `--includeFile` and `--excludeFile` we have
+  `--filename` and `--notFilename` respectively.
+  Also the semantics become consistent for such positive/negative filters.
diff --git a/doc/docgen.md b/doc/docgen.md
index 9e0a83bab..a05da8198 100644
--- a/doc/docgen.md
+++ b/doc/docgen.md
@@ -16,11 +16,26 @@ Introduction
 
 This document describes the `documentation generation tools`:idx: built into
 the [Nim compiler](nimc.html), which can generate HTML, Latex and JSON output
-from input ``.nim`` files and projects, as well as HTML and LaTeX from input RST
-(reStructuredText) files. The output documentation will include the module
+from input ``.nim`` files and projects.
+The output documentation will include the module
 dependencies (`import`), any top-level documentation comments (`##`), and
 exported symbols (`*`), including procedures, types, and variables.
 
+===================   ==============
+command               output format
+===================   ==============
+`nim doc`:cmd:        ``.html`` HTML
+`nim doc2tex`:cmd:    ``.tex`` LaTeX
+`nim jsondoc`:cmd:    ``.json`` JSON
+===================   ==============
+
+Nim can generate HTML and LaTeX from input Markdown and
+RST (reStructuredText) files as well, which is intended for writing
+standalone documents like user's guides and technical specifications.
+See [Nim-flavored Markdown and reStructuredText] document for the description
+of this feature and particularly section [Command line usage] for the full
+list of supported commands.
+
 Quick start
 -----------
 
@@ -319,6 +334,15 @@ external referencing requires to use ``.. importdoc:: <file>``
 directive to import `file` and to ensure that the corresponding
 ``.idx`` file was generated.
 
+Syntax for referencing is basically the same as for normal markup.
+Recall from [Referencing] that our parser supports two equivalent syntaxes
+for referencing, Markdown and RST one.
+So to reference ``proc f`` one should use something like that,
+depending on markup type:
+
+    Markdown                    RST
+
+    Ref. [proc f] or [f]        Ref. `proc f`_ or just f_ for a one-word case
 
 Nim local referencing
 ---------------------
@@ -327,10 +351,8 @@ You can reference Nim identifiers from Nim documentation comments
 inside their ``.nim`` file (or inside a ``.rst`` file included from
 a ``.nim``).
 This pertains to any exported symbol like `proc`, `const`, `iterator`, etc.
-Syntax for referencing is basically a normal RST one: addition of
-underscore `_` to a *link text*.
-Link text is either one word or a group of words enclosed by backticks `\``
-(for a one word case backticks are usually omitted).
+Link text is either one word or a group of words enclosed by delimiters
+(brackets ``[...]`` for Markdown or backticks `\`...\`_` for RST).
 Link text will be displayed *as is* while *link target* will be set to
 the anchor \[*] of Nim symbol that corresponds to link text.
 
diff --git a/doc/markdown_rst.md b/doc/markdown_rst.md
index a374749cb..9a266be9a 100644
--- a/doc/markdown_rst.md
+++ b/doc/markdown_rst.md
@@ -72,7 +72,7 @@ Features
 A large subset is implemented with some [limitations] and
 [additional Nim-specific features].
 
-Supported standard RST features:
+Supported common RST/Markdown features:
 
 * body elements
   + sections
@@ -83,10 +83,8 @@ Supported standard RST features:
     characters:  1. ... 2. ... *or* a. ... b. ... *or* A. ... B. ...
   + footnotes (including manually numbered, auto-numbered, auto-numbered
     with label, and auto-symbol footnotes) and citations
-  + definition lists
   + field lists
   + option lists
-  + indented literal blocks
   + quoted literal blocks
   + line blocks
   + simple tables
@@ -109,6 +107,33 @@ Supported standard RST features:
     (see [RST roles list] for description).
   + inline internal targets
 
+RST mode only features
+----------------------
+
++ RST syntax for definition lists (that is additional indentation after
+  a definition line)
++ indented literal blocks starting from ``::``
+
+Markdown-specific features
+--------------------------
+
+* Markdown tables
+* Markdown code blocks. For them the same additional arguments as for RST
+  code blocks can be provided (e.g. `test` or `number-lines`) but with
+  a one-line syntax like this:
+
+      ```nim test number-lines=10
+      echo "ok"
+      ```
+* Markdown links ``[...](...)``
+* Pandoc syntax for automatic links ``[...]``, see [Referencing] for description
++ Markdown literal blocks indented by 4 or more spaces
+* Markdown headlines
+* Markdown block quotes
+* Markdown syntax for definition lists
+* using ``1`` as auto-enumerator in enumerated lists like RST ``#``
+  (auto-enumerator ``1`` can not be used with ``#`` in the same list)
+
 Additional Nim-specific features
 --------------------------------
 
@@ -144,35 +169,16 @@ Additional Nim-specific features
 
   Here the dummy `//` will disappear, while options `compile`:option:
   and `doc`:option: will be left in the final document.
+* emoji / smiley symbols
 
 \[cmp:Sphinx] similar but different from the directives of
    Python [Sphinx directives] and [Sphinx roles] extensions
 
-Extra features
---------------
-
-Optional additional features, by default turned on:
-
-* emoji / smiley symbols
-* Markdown tables
-* Markdown code blocks. For them the same additional arguments as for RST
-  code blocks can be provided (e.g. `test` or `number-lines`) but with
-  a one-line syntax like this:
-
-      ```nim test number-lines=10
-      echo "ok"
-      ```
-* Markdown links
-* Markdown headlines
-* Markdown block quotes
-* using ``1`` as auto-enumerator in enumerated lists like RST ``#``
-  (auto-enumerator ``1`` can not be used with ``#`` in the same list)
-
 .. Note:: By default Nim has ``roSupportMarkdown`` and
    ``roSupportRawDirective`` turned **on**.
 
-.. warning:: Using Nim-specific features can cause other RST implementations
-  to fail on your document.
+.. warning:: Using Nim-specific features can cause other Markdown and
+  RST implementations to fail on your document.
 
 Referencing
 ===========