docgen: implement cross-document links (#20990)

* docgen: implement cross-document links

Fully implements https://github.com/nim-lang/RFCs/issues/125
Follow-up of: https://github.com/nim-lang/Nim/pull/18642 (for internal links)
and https://github.com/nim-lang/Nim/issues/20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering https://github.com/nim-lang/RFCs/issues/478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

2 files changed, 49 insertions, 44 deletions

diff --git a/nimdoc/testproject/expected/subdir/subdir_b/utils.html b/nimdoc/testproject/expected/subdir/subdir_b/utils.html
index 7d59e9e86..f0f30536f 100644
--- a/nimdoc/testproject/expected/subdir/subdir_b/utils.html
+++ b/nimdoc/testproject/expected/subdir/subdir_b/utils.html
@@ -226,7 +226,7 @@
                    cmp: proc (x: T; y: K): int {.closure.}): int" href="#binarySearch,openArray[T],K,proc(T,K)">binary search</a>.</p>
 <p>Note that <tt class="docutils literal"><span class="pre"><span class="Keyword">proc</span></span></tt> can be used in postfix form: <a class="reference internal nimdoc" title="proc binarySearch[T, K](a: openArray[T]; key: K;
                    cmp: proc (x: T; y: K): int {.closure.}): int" href="#binarySearch,openArray[T],K,proc(T,K)">binarySearch proc</a>.</p>
-<p>Ref. type like <a class="reference internal nimdoc" title="type G" href="#G">G</a> and <a class="reference internal nimdoc" title="type G" href="#G">type G</a> and <a class="reference internal nimdoc" title="type G" href="#G">G[T]</a> and <a class="reference internal nimdoc" title="type G" href="#G">type G*[T]</a>.</p>
+<p>Ref. type like <a class="reference internal nimdoc" title="object G" href="#G">G</a> and <a class="reference internal nimdoc" title="object G" href="#G">type G</a> and <a class="reference internal nimdoc" title="object G" href="#G">G[T]</a> and <a class="reference internal nimdoc" title="object G" href="#G">type G*[T]</a>.</p>
 <p>Group ref. with capital letters works: <a class="reference internal nimdoc" title="proc fN11 (2 overloads)" href="#fN11-procs-all">fN11</a> or <a class="reference internal nimdoc" title="proc fN11 (2 overloads)" href="#fN11-procs-all">fn11</a> </p>
 Ref. <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">[]</a> is the same as <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">proc `[]`(G[T])</a> because there are no overloads. The full form: <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">proc `[]`*[T](x: G[T]): T</a>Ref. <a class="reference internal nimdoc" title="proc `[]=`[T](a: var G[T]; index: int; value: T)" href="#[]=,G[T],int,T">[]=</a> aka <a class="reference internal nimdoc" title="proc `[]=`[T](a: var G[T]; index: int; value: T)" href="#[]=,G[T],int,T">`[]=`(G[T], int, T)</a>.Ref. <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">$</a> aka <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">proc $</a> or <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">proc `$`</a>.Ref. <a class="reference internal nimdoc" title="proc `$`[T](a: ref SomeType): string" href="#$,ref.SomeType">$(a: ref SomeType)</a>.Ref. <a class="reference internal nimdoc" title="iterator fooBar(a: seq[SomeType]): int" href="#fooBar.i,seq[SomeType]">foo_bar</a> aka <a class="reference internal nimdoc" title="iterator fooBar(a: seq[SomeType]): int" href="#fooBar.i,seq[SomeType]">iterator foo_bar_</a>.Ref. <a class="reference internal nimdoc" title="proc fn[T; U, V: SomeFloat]()" href="#fn">fn[T; U,V: SomeFloat]()</a>.Ref. <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">'big</a> or <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">func `'big`</a> or <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">`'big`(string)</a>.
 <h1><a class="toc-backref" id="pandoc-markdown" href="#pandoc-markdown">Pandoc Markdown</a></h1><p>Now repeat all the auto links of above in Pandoc Markdown Syntax.</p>
@@ -240,11 +240,11 @@ Ref. <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href=
                    cmp: proc (x: T; y: K): int {.closure.}): int" href="#binarySearch,openArray[T],K,proc(T,K)">binary search</a>.</p>
 <p>Note that <tt class="docutils literal"><span class="pre"><span class="Keyword">proc</span></span></tt> can be used in postfix form: <a class="reference internal nimdoc" title="proc binarySearch[T, K](a: openArray[T]; key: K;
                    cmp: proc (x: T; y: K): int {.closure.}): int" href="#binarySearch,openArray[T],K,proc(T,K)">binarySearch proc</a>.</p>
-<p>Ref. type like <a class="reference internal nimdoc" title="type G" href="#G">G</a> and <a class="reference internal nimdoc" title="type G" href="#G">type G</a> and <a class="reference internal nimdoc" title="type G" href="#G">G[T]</a> and <a class="reference internal nimdoc" title="type G" href="#G">type G*[T]</a>.</p>
+<p>Ref. type like <a class="reference internal nimdoc" title="object G" href="#G">G</a> and <a class="reference internal nimdoc" title="object G" href="#G">type G</a> and <a class="reference internal nimdoc" title="object G" href="#G">G[T]</a> and <a class="reference internal nimdoc" title="object G" href="#G">type G*[T]</a>.</p>
 <p>Group ref. with capital letters works: <a class="reference internal nimdoc" title="proc fN11 (2 overloads)" href="#fN11-procs-all">fN11</a> or <a class="reference internal nimdoc" title="proc fN11 (2 overloads)" href="#fN11-procs-all">fn11</a></p>
 <p>Ref. <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">`[]`</a> is the same as <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">proc `[]`(G[T])</a> because there are no overloads. The full form: <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">proc `[]`*[T](x: G[T]): T</a> Ref. <a class="reference internal nimdoc" title="proc `[]=`[T](a: var G[T]; index: int; value: T)" href="#[]=,G[T],int,T">`[]=`</a> aka <a class="reference internal nimdoc" title="proc `[]=`[T](a: var G[T]; index: int; value: T)" href="#[]=,G[T],int,T">`[]=`(G[T], int, T)</a>. Ref. <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">$</a> aka <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">proc $</a> or <a class="reference internal nimdoc" title="proc $ (2 overloads)" href="#$-procs-all">proc `$`</a>. Ref. <a class="reference internal nimdoc" title="proc `$`[T](a: ref SomeType): string" href="#$,ref.SomeType">$(a: ref SomeType)</a>. Ref. <a class="reference internal nimdoc" title="iterator fooBar(a: seq[SomeType]): int" href="#fooBar.i,seq[SomeType]">foo_bar</a> aka <a class="reference internal nimdoc" title="iterator fooBar(a: seq[SomeType]): int" href="#fooBar.i,seq[SomeType]">iterator foo_bar_</a>. Ref. <a class="reference internal nimdoc" title="proc fn[T; U, V: SomeFloat]()" href="#fn">fn[T; U,V: SomeFloat]()</a>. Ref. <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">'big</a> or <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">func `'big`</a> or <a class="reference internal nimdoc" title="proc `'big`(a: string): SomeType" href="#'big,string">`'big`(string)</a>.</p>
 
-<h2><a class="toc-backref" id="pandoc-markdown-link-name-syntax" href="#pandoc-markdown-link-name-syntax">Link name syntax</a></h2><p>Pandoc Markdown has synax for changing text of links: Ref. <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">this proc</a> or <a class="reference internal nimdoc" title="type G" href="#G">another symbol</a>.</p>
+<h2><a class="toc-backref" id="pandoc-markdown-link-name-syntax" href="#pandoc-markdown-link-name-syntax">Link name syntax</a></h2><p>Pandoc Markdown has synax for changing text of links: Ref. <a class="reference internal nimdoc" title="proc `[]`[T](x: G[T]): T" href="#[],G[T]">this proc</a> or <a class="reference internal nimdoc" title="object G" href="#G">another symbol</a>.</p>
 
 <h2><a class="toc-backref" id="pandoc-markdown-symbols-documentation" href="#pandoc-markdown-symbols-documentation">Symbols documentation</a></h2><p>Let us repeat auto links from symbols section below:</p>
 <p>There is also variant <a class="reference internal nimdoc" title="proc f(x: G[string])" href="#f,G[string]">f(G[string])</a>. See also <a class="reference internal nimdoc" title="proc f(x: G[int])" href="#f,G[int]">f(G[int])</a>.</p>
diff --git a/nimdoc/testproject/expected/subdir/subdir_b/utils.idx b/nimdoc/testproject/expected/subdir/subdir_b/utils.idx
index 007101b37..c2f148a92 100644
--- a/nimdoc/testproject/expected/subdir/subdir_b/utils.idx
+++ b/nimdoc/testproject/expected/subdir/subdir_b/utils.idx
@@ -1,41 +1,46 @@
-funWithGenerics	subdir/subdir_b/utils.html#funWithGenerics,T,U	utils: funWithGenerics[T, U: SomeFloat](a: T; b: U)	
-enumValueA	subdir/subdir_b/utils.html#enumValueA	SomeType.enumValueA	
-enumValueB	subdir/subdir_b/utils.html#enumValueB	SomeType.enumValueB	
-enumValueC	subdir/subdir_b/utils.html#enumValueC	SomeType.enumValueC	
-SomeType	subdir/subdir_b/utils.html#SomeType	utils: SomeType	
-G	subdir/subdir_b/utils.html#G	utils: G	
-someType	subdir/subdir_b/utils.html#someType_2	utils: someType(): SomeType	
-fn2	subdir/subdir_b/utils.html#fn2	utils: fn2()	
-fn2	subdir/subdir_b/utils.html#fn2,int	utils: fn2(x: int)	
-fn2	subdir/subdir_b/utils.html#fn2,int,float	utils: fn2(x: int; y: float)	
-binarySearch	subdir/subdir_b/utils.html#binarySearch,openArray[T],K,proc(T,K)	utils: binarySearch[T, K](a: openArray[T]; key: K;\n                   cmp: proc (x: T; y: K): int {.closure.}): int	
-fn3	subdir/subdir_b/utils.html#fn3	utils: fn3(): auto	
-fn4	subdir/subdir_b/utils.html#fn4	utils: fn4(): auto	
-fn5	subdir/subdir_b/utils.html#fn5	utils: fn5()	
-fn6	subdir/subdir_b/utils.html#fn6	utils: fn6()	
-fn7	subdir/subdir_b/utils.html#fn7	utils: fn7()	
-fn8	subdir/subdir_b/utils.html#fn8	utils: fn8(): auto	
-fn9	subdir/subdir_b/utils.html#fn9,int	utils: fn9(a: int): int	
-fn10	subdir/subdir_b/utils.html#fn10,int	utils: fn10(a: int): int	
-fN11	subdir/subdir_b/utils.html#fN11	utils: fN11()	
-fN11	subdir/subdir_b/utils.html#fN11,int	utils: fN11(x: int)	
-aEnum	subdir/subdir_b/utils.html#aEnum.t	utils: aEnum(): untyped	
-bEnum	subdir/subdir_b/utils.html#bEnum.t	utils: bEnum(): untyped	
-fromUtilsGen	subdir/subdir_b/utils.html#fromUtilsGen.t	utils: fromUtilsGen(): untyped	
-f	subdir/subdir_b/utils.html#f,G[int]	utils: f(x: G[int])	
-f	subdir/subdir_b/utils.html#f,G[string]	utils: f(x: G[string])	
-`[]`	subdir/subdir_b/utils.html#[],G[T]	utils: `[]`[T](x: G[T]): T	
-`[]=`	subdir/subdir_b/utils.html#[]=,G[T],int,T	utils: `[]=`[T](a: var G[T]; index: int; value: T)	
-`$`	subdir/subdir_b/utils.html#$,G[T]	utils: `$`[T](a: G[T]): string	
-`$`	subdir/subdir_b/utils.html#$,ref.SomeType	utils: `$`[T](a: ref SomeType): string	
-fooBar	subdir/subdir_b/utils.html#fooBar.i,seq[SomeType]	utils: fooBar(a: seq[SomeType]): int	
-fn	subdir/subdir_b/utils.html#fn	utils: fn[T; U, V: SomeFloat]()	
-`'big`	subdir/subdir_b/utils.html#'big,string	utils: `&apos;big`(a: string): SomeType	
-This is now a header	subdir/subdir_b/utils.html#this-is-now-a-header	 This is now a header	
-Next header	subdir/subdir_b/utils.html#this-is-now-a-header-next-header	  Next header	
-And so on	subdir/subdir_b/utils.html#next-header-and-so-on	   And so on	
-More headers	subdir/subdir_b/utils.html#more-headers	 More headers	
-Up to level 6	subdir/subdir_b/utils.html#more-headers-up-to-level-6	      Up to level 6	
-Pandoc Markdown	subdir/subdir_b/utils.html#pandoc-markdown	 Pandoc Markdown	
-Link name syntax	subdir/subdir_b/utils.html#pandoc-markdown-link-name-syntax	  Link name syntax	
-Symbols documentation	subdir/subdir_b/utils.html#pandoc-markdown-symbols-documentation	  Symbols documentation	
+nimTitle	utils	subdir/subdir_b/utils.html	module subdir/subdir_b/utils		0
+nim	funWithGenerics	subdir/subdir_b/utils.html#funWithGenerics,T,U	proc funWithGenerics[T, U: SomeFloat](a: T; b: U)		1
+nim	enumValueA	subdir/subdir_b/utils.html#enumValueA	SomeType.enumValueA		45
+nim	enumValueB	subdir/subdir_b/utils.html#enumValueB	SomeType.enumValueB		45
+nim	enumValueC	subdir/subdir_b/utils.html#enumValueC	SomeType.enumValueC		45
+nim	SomeType	subdir/subdir_b/utils.html#SomeType	enum SomeType		45
+nim	G	subdir/subdir_b/utils.html#G	object G		49
+nim	someType	subdir/subdir_b/utils.html#someType_2	proc someType(): SomeType		52
+nim	fn2	subdir/subdir_b/utils.html#fn2	proc fn2()		57
+nim	fn2	subdir/subdir_b/utils.html#fn2,int	proc fn2(x: int)		58
+nim	fn2	subdir/subdir_b/utils.html#fn2,int,float	proc fn2(x: int; y: float)		61
+nim	binarySearch	subdir/subdir_b/utils.html#binarySearch,openArray[T],K,proc(T,K)	proc binarySearch[T, K](a: openArray[T]; key: K;\n                   cmp: proc (x: T; y: K): int {.closure.}): int		63
+nim	fn3	subdir/subdir_b/utils.html#fn3	proc fn3(): auto		66
+nim	fn4	subdir/subdir_b/utils.html#fn4	proc fn4(): auto		67
+nim	fn5	subdir/subdir_b/utils.html#fn5	proc fn5()		69
+nim	fn6	subdir/subdir_b/utils.html#fn6	proc fn6()		70
+nim	fn7	subdir/subdir_b/utils.html#fn7	proc fn7()		72
+nim	fn8	subdir/subdir_b/utils.html#fn8	proc fn8(): auto		75
+nim	fn9	subdir/subdir_b/utils.html#fn9,int	proc fn9(a: int): int		78
+nim	fn10	subdir/subdir_b/utils.html#fn10,int	proc fn10(a: int): int		79
+nim	fN11	subdir/subdir_b/utils.html#fN11	proc fN11()		85
+nim	fN11	subdir/subdir_b/utils.html#fN11,int	proc fN11(x: int)		86
+nim	aEnum	subdir/subdir_b/utils.html#aEnum.t	template aEnum(): untyped		90
+nim	bEnum	subdir/subdir_b/utils.html#bEnum.t	template bEnum(): untyped		95
+nim	fromUtilsGen	subdir/subdir_b/utils.html#fromUtilsGen.t	template fromUtilsGen(): untyped		106
+nim	f	subdir/subdir_b/utils.html#f,G[int]	proc f(x: G[int])		130
+nim	f	subdir/subdir_b/utils.html#f,G[string]	proc f(x: G[string])		133
+nim	`[]`	subdir/subdir_b/utils.html#[],G[T]	proc `[]`[T](x: G[T]): T		140
+nim	`[]=`	subdir/subdir_b/utils.html#[]=,G[T],int,T	proc `[]=`[T](a: var G[T]; index: int; value: T)		144
+nim	`$`	subdir/subdir_b/utils.html#$,G[T]	proc `$`[T](a: G[T]): string		148
+nim	`$`	subdir/subdir_b/utils.html#$,ref.SomeType	proc `$`[T](a: ref SomeType): string		152
+nim	fooBar	subdir/subdir_b/utils.html#fooBar.i,seq[SomeType]	iterator fooBar(a: seq[SomeType]): int		156
+nim	fn	subdir/subdir_b/utils.html#fn	proc fn[T; U, V: SomeFloat]()		160
+nim	`'big`	subdir/subdir_b/utils.html#'big,string	proc `'big`(a: string): SomeType		164
+nimgrp	$	subdir/subdir_b/utils.html#$-procs-all	proc		148
+nimgrp	fn11	subdir/subdir_b/utils.html#fN11-procs-all	proc		85
+nimgrp	fn2	subdir/subdir_b/utils.html#fn2-procs-all	proc		57
+nimgrp	f	subdir/subdir_b/utils.html#f-procs-all	proc		130
+heading	This is now a header	subdir/subdir_b/utils.html#this-is-now-a-header	 This is now a header		0
+heading	Next header	subdir/subdir_b/utils.html#this-is-now-a-header-next-header	  Next header		0
+heading	And so on	subdir/subdir_b/utils.html#next-header-and-so-on	   And so on		0
+heading	More headers	subdir/subdir_b/utils.html#more-headers	 More headers		0
+heading	Up to level 6	subdir/subdir_b/utils.html#more-headers-up-to-level-6	      Up to level 6		0
+heading	Pandoc Markdown	subdir/subdir_b/utils.html#pandoc-markdown	 Pandoc Markdown		0
+heading	Link name syntax	subdir/subdir_b/utils.html#pandoc-markdown-link-name-syntax	  Link name syntax		0
+heading	Symbols documentation	subdir/subdir_b/utils.html#pandoc-markdown-symbols-documentation	  Symbols documentation		0