Extract markdown/rst doc into separate file (#20404)

* Extract Markdown & Rst doc into separate file

This documentation should be extracted into separate file
as it's user's documentation, which can be used as a separate
utility for compiling `.md/.rst` files.

* Restructure: move markup info into markdown_rst.md

+Markdown link migration

2 files changed, 263 insertions, 26 deletions

diff --git a/doc/docgen.md b/doc/docgen.md
index 5e03f92ff..268087cfc 100644
--- a/doc/docgen.md
+++ b/doc/docgen.md
@@ -20,16 +20,6 @@ from input ``.nim`` files and projects, as well as HTML and LaTeX from input RST
 dependencies (`import`), any top-level documentation comments (`##`), and
 exported symbols (`*`), including procedures, types, and variables.
 
-===================   ======================   ============   ==============
-command               runs on...               input format   output format
-===================   ======================   ============   ==============
-`nim doc`:cmd:        documentation comments   ``.nim``       ``.html`` HTML
-`nim doc2tex`:cmd:    ″                        ″              ``.tex`` LaTeX
-`nim jsondoc`:cmd:    ″                        ″              ``.json`` JSON
-`nim rst2html`:cmd:   standalone rst files     ``.rst``       ``.html`` HTML
-`nim rst2tex`:cmd:    ″                        ″              ``.tex`` LaTeX
-===================   ======================   ============   ==============
-
 Quick start
 -----------
 
@@ -490,18 +480,8 @@ highlighting with the ``.. code-block:: nim`` prefix. ``code-block`` also
 supports highlighting of a few other languages supported by the
 [packages/docutils/highlite module](highlite.html).
 
-Usage:
-
-  ```cmd
-  nim rst2html docgen.rst
-  ```
-
-Output::
-  You're reading it!
-
-The `rst2tex`:option: command is invoked identically to `rst2html`:option:,
-but outputs a ``.tex`` file instead of ``.html``.
-
+See [Markdown and RST markup languages](markdown_rst.html) for
+usage of those commands.
 
 HTML anchor generation
 ======================
@@ -628,10 +608,9 @@ Additional resources
 
 * [Nim Compiler User Guide](nimc.html#compiler-usage-commandminusline-switches)
 
-* Documentation for [rst module](rst.html) -- Nim RST/Markdown parser.
-
-* [RST Quick Reference](
-  http://docutils.sourceforge.net/docs/user/rst/quickref.html)
+* already mentioned documentation for
+  [Markdown and RST markup languages](markdown_rst.html), which also
+  contains the list of implemented features of these markup languages.
 
 The output for HTML and LaTeX comes from the ``config/nimdoc.cfg`` and
 ``config/nimdoc.tex.cfg`` configuration files. You can add and modify these
diff --git a/doc/markdown_rst.md b/doc/markdown_rst.md
new file mode 100644
index 000000000..c507f2511
--- /dev/null
+++ b/doc/markdown_rst.md
@@ -0,0 +1,258 @@
+==========================================
+Nim-flavored Markdown and reStructuredText
+==========================================
+
+:Author: Andrey Makarov
+:Version: |nimversion|
+
+.. default-role:: code
+.. include:: rstcommon.rst
+.. contents::
+
+Both `Markdown`:idx: (md) and `reStructuredText`:idx: (RST) are markup
+languages whose goal is to typeset texts with complex structure,
+formatting and references using simple plaintext representation.
+
+Command line usage
+==================
+
+Usage (to convert Markdown into HTML):
+
+  ```cmd
+  nim md2html markdown_rst.md
+  ```
+
+Output::
+  You're reading it!
+
+The `md2tex`:option: command is invoked identically to `md2html`:option:,
+but outputs a ``.tex`` file instead of ``.html``.
+
+These tools embedded into Nim compiler; the compiler can output
+the result to HTML \[#html] or Latex \[#latex].
+
+\[#html] commands `nim doc`:cmd: for ``*.nim`` files and
+   `nim rst2html`:cmd: for ``*.rst`` files
+
+\[#latex] commands `nim doc2tex`:cmd: for ``*.nim`` and
+   `nim rst2tex`:cmd: for ``*.rst``.
+
+Full list of supported commands:
+
+===================   ======================   ============   ==============
+command               runs on...               input format   output format
+===================   ======================   ============   ==============
+`nim md2html`:cmd:    standalone md files      ``.md``        ``.html`` HTML
+`nim md2tex`:cmd:     same                     same           ``.tex`` LaTeX
+`nim rst2html`:cmd:   standalone rst files     ``.rst``       ``.html`` HTML
+`nim rst2tex`:cmd:    same                     same           ``.tex`` LaTeX
+`nim doc`:cmd:        documentation comments   ``.nim``       ``.html`` HTML
+`nim doc2tex`:cmd:    same                     same           ``.tex`` LaTeX
+`nim jsondoc`:cmd:    same                     same           ``.json`` JSON
+===================   ======================   ============   ==============
+
+
+Basic markup
+============
+
+If you are new to Markdown/RST please consider reading the following:
+
+1) [Markdown Basic Syntax]
+2) a long specification of Markdown: [CommonMark Spec]
+3) a short [quick introduction] to RST
+4) an [RST reference]: a comprehensive cheatsheet for RST
+5) a more formal 50-page [RST specification].
+
+Features
+--------
+
+A large subset is implemented with some [limitations] and
+[additional Nim-specific features].
+
+Supported standard RST features:
+
+* body elements
+  + sections
+  + transitions
+  + paragraphs
+  + bullet lists using \+, \*, \-
+  + enumerated lists using arabic numerals or alphabet
+    characters:  1. ... 2. ... *or* a. ... b. ... *or* A. ... B. ...
+  + footnotes (including manually numbered, auto-numbered, auto-numbered
+    with label, and auto-symbol footnotes) and citations
+  + definition lists
+  + field lists
+  + option lists
+  + indented literal blocks
+  + quoted literal blocks
+  + line blocks
+  + simple tables
+  + directives (see official documentation in [RST directives list]):
+    - ``image``, ``figure`` for including images and videos
+    - ``code``
+    - ``contents`` (table of contents), ``container``, ``raw``
+    - ``include``
+    - admonitions: "attention", "caution", "danger", "error", "hint",
+      "important", "note", "tip", "warning", "admonition"
+    - substitution definitions: `replace` and `image`
+  + comments
+* inline markup
+  + *emphasis*, **strong emphasis**,
+    ``inline literals``, hyperlink references (including embedded URI),
+    substitution references, standalone hyperlinks,
+    internal links (inline and outline)
+  + \`interpreted text\` with roles ``:literal:``, ``:strong:``,
+    ``emphasis``, ``:sub:``/``:subscript:``, ``:sup:``/``:superscript:``
+    (see [RST roles list] for description).
+  + inline internal targets
+
+Additional Nim-specific features
+--------------------------------
+
+* directives: ``code-block`` \[cmp:Sphinx], ``title``,
+  ``index`` \[cmp:Sphinx]
+* predefined roles
+  - ``:nim:`` (default), ``:c:`` (C programming language),
+    ``:python:``, ``:yaml:``, ``:java:``, ``:cpp:`` (C++), ``:csharp`` (C#).
+    That is every language that [highlite](highlite.html) supports.
+    They turn on appropriate syntax highlighting in inline code.
+
+    .. Note:: default role for Nim files is ``:nim:``,
+              for ``*.rst`` it's currently ``:literal:``.
+
+  - generic command line highlighting roles:
+    - ``:cmd:`` for commands and common shells syntax
+    - ``:console:`` the same  for interactive sessions
+      (commands should be prepended by ``$``)
+    - ``:program:`` for executable names \[cmp:Sphinx]
+      (one can just use ``:cmd:`` on single word)
+    - ``:option:`` for command line options \[cmp:Sphinx]
+  - ``:tok:``, a role for highlighting of programming language tokens
+* ***triple emphasis*** (bold and italic) using \*\*\*
+* ``:idx:`` role for \`interpreted text\` to include the link to this
+  text into an index (example: [Nim index]).
+* double slash `//` in option lists serves as a prefix for any option that
+  starts from a word (without any leading symbols like `-`, `--`, `/`)::
+
+    //compile   compile the project
+    //doc       generate documentation
+
+  Here the dummy `//` will disappear, while options `compile`:option:
+  and `doc`:option: will be left in the final document.
+
+\[cmp:Sphinx] similar but different from the directives of
+   Python [Sphinx directives] and [Sphinx roles] extensions
+
+Extra features
+--------------
+
+Optional additional features, by default turned on:
+
+* emoji / smiley symbols
+* Markdown tables
+* Markdown code blocks. For them the same additional arguments as for RST
+  code blocks can be provided (e.g. `test` or `number-lines`) but with
+  a one-line syntax like this::
+
+    ```nim test number-lines=10
+    echo "ok"
+    ```
+* Markdown links
+* Markdown headlines
+* Markdown block quotes
+* using ``1`` as auto-enumerator in enumerated lists like RST ``#``
+  (auto-enumerator ``1`` can not be used with ``#`` in the same list)
+
+.. Note:: By default Nim has ``roSupportMarkdown`` and
+   ``roSupportRawDirective`` turned **on**.
+
+.. warning:: Using Nim-specific features can cause other RST implementations
+  to fail on your document.
+
+Idiosyncrasies
+--------------
+
+Currently we do **not** aim at 100% Markdown or RST compatibility in inline
+markup recognition rules because that would provide very little user value.
+This parser has 2 modes for inline markup:
+
+1) Markdown-like mode which is enabled by `roPreferMarkdown` option
+   (turned **on** by default).
+
+   .. Note:: RST features like directives are still turned **on**
+
+2) Compatibility mode which is RST rules.
+
+.. Note:: in both modes the parser interpretes text between single
+     backticks (code) identically:
+     backslash does not escape; the only exception: ``\`` folowed by `
+     does escape so that we can always input a single backtick ` in
+     inline code. However that makes impossible to input code with
+     ``\`` at the end in *single* backticks, one must use *double*
+     backticks::
+
+       `\`   -- WRONG
+       ``\`` -- GOOD
+       So single backticks can always be input: `\`` will turn to ` code
+
+.. Attention::
+   We don't support some obviously poor design choices of Markdown (or RST).
+
+   - no support for the rule of 2 spaces causing a line break in Markdown
+     (use RST "line blocks" syntax for making line breaks)
+
+   - interpretation of Markdown block quotes is also slightly different,
+     e.g. case
+
+     ::
+
+       >>> foo
+       > bar
+       >>baz
+
+     is a single 3rd-level quote `foo bar baz` in original Markdown, while
+     in Nim we naturally see it as 3rd-level quote `foo` + 1st level `bar` +
+     2nd level `baz`:
+
+     >>> foo
+     > bar
+     >>baz
+
+Limitations
+-----------
+
+* no Unicode support in character width calculations
+* body elements
+  - no roman numerals in enumerated lists
+  - no doctest blocks
+  - no grid tables
+  - some directives are missing (check official [RST directives list]):
+    ``parsed-literal``, ``sidebar``, ``topic``, ``math``, ``rubric``,
+    ``epigraph``, ``highlights``, ``pull-quote``, ``compound``,
+    ``table``, ``csv-table``, ``list-table``, ``section-numbering``,
+    ``header``, ``footer``, ``meta``, ``class``
+    - no ``role`` directives and no custom interpreted text roles
+    - some standard roles are not supported (check [RST roles list])
+    - no generic admonition support
+* inline markup
+  - no simple-inline-markup
+  - no embedded aliases
+
+Additional resources
+--------------------
+
+* See [Nim DocGen Tools Guide](docgen.html) for the details about
+  `nim doc`:cmd: command and idiosyncrasies of documentation markup in
+  ``.nim`` files and Nim programming language projects.
+* See also documentation for [rst module](rst.html) -- Nim RST/Markdown parser.
+
+.. _Markdown Basic Syntax: https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax
+.. _CommonMark Spec: https://spec.commonmark.org/0.30
+.. _quick introduction: https://docutils.sourceforge.io/docs/user/rst/quickstart.html
+.. _RST reference: https://docutils.sourceforge.io/docs/user/rst/quickref.html
+.. _RST specification: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html
+.. _RST directives list: https://docutils.sourceforge.io/docs/ref/rst/directives.html
+.. _RST roles list: https://docutils.sourceforge.io/docs/ref/rst/roles.html
+.. _Nim index: https://nim-lang.org/docs/theindex.html
+.. _Sphinx directives: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html
+.. _Sphinx roles: https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html