From 2620da9bf9994d09da3d5b31b29c95855fb79a41 Mon Sep 17 00:00:00 2001 From: Andrey Makarov Date: Wed, 4 Jan 2023 23:19:01 +0300 Subject: docgen: implement cross-document links (#20990) * docgen: implement cross-document links Fully implements https://github.com/nim-lang/RFCs/issues/125 Follow-up of: https://github.com/nim-lang/Nim/pull/18642 (for internal links) and https://github.com/nim-lang/Nim/issues/20127. Overview -------- Explicit import-like directive is required, called `.. importdoc::`. (the syntax is % RST, Markdown will use it for a while). Then one can reference any symbols/headings/anchors, as if they were in the local file (but they will be prefixed with a module name or markup document in link text). It's possible to reference anything from anywhere (any direction in `.nim`/`.md`/`.rst` files). See `doc/docgen.md` for full description. Working is based on `.idx` files, hence one needs to generate all `.idx` beforehand. A dedicated option `--index:only` is introduced (and a separate stage for `--index:only` is added to `kochdocs.nim`). Performance note ---------------- Full run for `./koch docs` now takes 185% of the time before this PR. (After: 315 s, before: 170 s on my PC). All the time seems to be spent on `--index:only` run, which takes almost as much (85%) of normal doc run -- it seems that most time is spent on file parsing, turning off HTML generation phase has not helped much. (One could avoid it by specifying list of files that can be referenced and pre-processing only them. But it can become error-prone and I assume that these linke will be **everywhere** in the repository anyway, especially considering https://github.com/nim-lang/RFCs/issues/478. So every `.nim`/`.md` file is processed for `.idx` first). But that's all without significant part of repository converted to cross-module auto links. To estimate impact I checked the time for `doc`ing a few files (after all indexes have been generated), and everywhere difference was **negligible**. E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large `os.idx` and hence should have been a case with relatively large performance impact, but: * After: 0.59 s. * Before: 0.59 s. So Nim compiler works so slow that doc part basically does not matter :-) Testing ------- 1) added `extlinks` test to `nimdoc/` 2) checked that `theindex.html` is still correct 2) fixed broken auto-links for modules that were derived from `os.nim` by adding appropriate ``importdoc`` Implementation note ------------------- Parsing and formating of `.idx` entries is moved into a dedicated `rstidx.nim` module from `rstgen.nim`. `.idx` file format changed: * fields are not escaped in most cases because we need original strings for referencing, not HTML ones (the exception is linkTitle for titles and headings). Escaping happens later -- on the stage of `rstgen` buildIndex, etc. * all lines have fixed number of columns 6 * added discriminator tag as a first column, it always allows distinguish Nim/markup entries, titles/headings, etc. `rstgen` does not rely any more (in most cases) on ad-hoc logic to determine what type each entry is. * there is now always a title entry added at the first line. * add a line number as 6th column * linkTitle (4th) column has a different format: before it was like `module: funcName()`, now it's `proc funcName()`. (This format is also propagated to `theindex.html` and search results, I kept it that way since I like it more though it's discussible.) This column is what used for Nim symbols resolution. * also changed details on column format for headings and titles: "keyword" is original, "linkTitle" is HTML one * fix paths on Windows + more clear code * Update compiler/docgen.nim Co-authored-by: Andreas Rumpf * Handle .md and .nim paths uniformly in findRefFile * handle titles better + more comments * don't allow markup overwrite index title for .nim files Co-authored-by: Andreas Rumpf --- .../expected/subdir/subdir_b/utils.html | 6 +- .../testproject/expected/subdir/subdir_b/utils.idx | 87 ++++++++++++---------- 2 files changed, 49 insertions(+), 44 deletions(-) (limited to 'nimdoc/testproject/expected/subdir/subdir_b') diff --git a/nimdoc/testproject/expected/subdir/subdir_b/utils.html b/nimdoc/testproject/expected/subdir/subdir_b/utils.html index 7d59e9e86..f0f30536f 100644 --- a/nimdoc/testproject/expected/subdir/subdir_b/utils.html +++ b/nimdoc/testproject/expected/subdir/subdir_b/utils.html @@ -226,7 +226,7 @@ cmp: proc (x: T; y: K): int {.closure.}): int" href="#binarySearch,openArray[T],K,proc(T,K)">binary search.

Note that proc can be used in postfix form: binarySearch proc.

-

Ref. type like G and type G and G[T] and type G*[T].

+

Ref. type like G and type G and G[T] and type G*[T].

Group ref. with capital letters works: fN11 or fn11

Ref. [] is the same as proc `[]`(G[T]) because there are no overloads. The full form: proc `[]`*[T](x: G[T]): TRef. []= aka `[]=`(G[T], int, T).Ref. $ aka proc $ or proc `$`.Ref. $(a: ref SomeType).Ref. foo_bar aka iterator foo_bar_.Ref. fn[T; U,V: SomeFloat]().Ref. 'big or func `'big` or `'big`(string).

Pandoc Markdown

Now repeat all the auto links of above in Pandoc Markdown Syntax.

@@ -240,11 +240,11 @@ Ref. binary search.

Note that proc can be used in postfix form: binarySearch proc.

-

Ref. type like G and type G and G[T] and type G*[T].

+

Ref. type like G and type G and G[T] and type G*[T].

Group ref. with capital letters works: fN11 or fn11

Ref. `[]` is the same as proc `[]`(G[T]) because there are no overloads. The full form: proc `[]`*[T](x: G[T]): T Ref. `[]=` aka `[]=`(G[T], int, T). Ref. $ aka proc $ or proc `$`. Ref. $(a: ref SomeType). Ref. foo_bar aka iterator foo_bar_. Ref. fn[T; U,V: SomeFloat](). Ref. 'big or func `'big` or `'big`(string).

-

Link name syntax

Pandoc Markdown has synax for changing text of links: Ref. this proc or another symbol.

+

Link name syntax

Pandoc Markdown has synax for changing text of links: Ref. this proc or another symbol.

Symbols documentation

Let us repeat auto links from symbols section below:

There is also variant f(G[string]). See also f(G[int]).

diff --git a/nimdoc/testproject/expected/subdir/subdir_b/utils.idx b/nimdoc/testproject/expected/subdir/subdir_b/utils.idx index 007101b37..c2f148a92 100644 --- a/nimdoc/testproject/expected/subdir/subdir_b/utils.idx +++ b/nimdoc/testproject/expected/subdir/subdir_b/utils.idx @@ -1,41 +1,46 @@ -funWithGenerics subdir/subdir_b/utils.html#funWithGenerics,T,U utils: funWithGenerics[T, U: SomeFloat](a: T; b: U) -enumValueA subdir/subdir_b/utils.html#enumValueA SomeType.enumValueA -enumValueB subdir/subdir_b/utils.html#enumValueB SomeType.enumValueB -enumValueC subdir/subdir_b/utils.html#enumValueC SomeType.enumValueC -SomeType subdir/subdir_b/utils.html#SomeType utils: SomeType -G subdir/subdir_b/utils.html#G utils: G -someType subdir/subdir_b/utils.html#someType_2 utils: someType(): SomeType -fn2 subdir/subdir_b/utils.html#fn2 utils: fn2() -fn2 subdir/subdir_b/utils.html#fn2,int utils: fn2(x: int) -fn2 subdir/subdir_b/utils.html#fn2,int,float utils: fn2(x: int; y: float) -binarySearch subdir/subdir_b/utils.html#binarySearch,openArray[T],K,proc(T,K) utils: binarySearch[T, K](a: openArray[T]; key: K;\n cmp: proc (x: T; y: K): int {.closure.}): int -fn3 subdir/subdir_b/utils.html#fn3 utils: fn3(): auto -fn4 subdir/subdir_b/utils.html#fn4 utils: fn4(): auto -fn5 subdir/subdir_b/utils.html#fn5 utils: fn5() -fn6 subdir/subdir_b/utils.html#fn6 utils: fn6() -fn7 subdir/subdir_b/utils.html#fn7 utils: fn7() -fn8 subdir/subdir_b/utils.html#fn8 utils: fn8(): auto -fn9 subdir/subdir_b/utils.html#fn9,int utils: fn9(a: int): int -fn10 subdir/subdir_b/utils.html#fn10,int utils: fn10(a: int): int -fN11 subdir/subdir_b/utils.html#fN11 utils: fN11() -fN11 subdir/subdir_b/utils.html#fN11,int utils: fN11(x: int) -aEnum subdir/subdir_b/utils.html#aEnum.t utils: aEnum(): untyped -bEnum subdir/subdir_b/utils.html#bEnum.t utils: bEnum(): untyped -fromUtilsGen subdir/subdir_b/utils.html#fromUtilsGen.t utils: fromUtilsGen(): untyped -f subdir/subdir_b/utils.html#f,G[int] utils: f(x: G[int]) -f subdir/subdir_b/utils.html#f,G[string] utils: f(x: G[string]) -`[]` subdir/subdir_b/utils.html#[],G[T] utils: `[]`[T](x: G[T]): T -`[]=` subdir/subdir_b/utils.html#[]=,G[T],int,T utils: `[]=`[T](a: var G[T]; index: int; value: T) -`$` subdir/subdir_b/utils.html#$,G[T] utils: `$`[T](a: G[T]): string -`$` subdir/subdir_b/utils.html#$,ref.SomeType utils: `$`[T](a: ref SomeType): string -fooBar subdir/subdir_b/utils.html#fooBar.i,seq[SomeType] utils: fooBar(a: seq[SomeType]): int -fn subdir/subdir_b/utils.html#fn utils: fn[T; U, V: SomeFloat]() -`'big` subdir/subdir_b/utils.html#'big,string utils: `'big`(a: string): SomeType -This is now a header subdir/subdir_b/utils.html#this-is-now-a-header This is now a header -Next header subdir/subdir_b/utils.html#this-is-now-a-header-next-header Next header -And so on subdir/subdir_b/utils.html#next-header-and-so-on And so on -More headers subdir/subdir_b/utils.html#more-headers More headers -Up to level 6 subdir/subdir_b/utils.html#more-headers-up-to-level-6 Up to level 6 -Pandoc Markdown subdir/subdir_b/utils.html#pandoc-markdown Pandoc Markdown -Link name syntax subdir/subdir_b/utils.html#pandoc-markdown-link-name-syntax Link name syntax -Symbols documentation subdir/subdir_b/utils.html#pandoc-markdown-symbols-documentation Symbols documentation +nimTitle utils subdir/subdir_b/utils.html module subdir/subdir_b/utils 0 +nim funWithGenerics subdir/subdir_b/utils.html#funWithGenerics,T,U proc funWithGenerics[T, U: SomeFloat](a: T; b: U) 1 +nim enumValueA subdir/subdir_b/utils.html#enumValueA SomeType.enumValueA 45 +nim enumValueB subdir/subdir_b/utils.html#enumValueB SomeType.enumValueB 45 +nim enumValueC subdir/subdir_b/utils.html#enumValueC SomeType.enumValueC 45 +nim SomeType subdir/subdir_b/utils.html#SomeType enum SomeType 45 +nim G subdir/subdir_b/utils.html#G object G 49 +nim someType subdir/subdir_b/utils.html#someType_2 proc someType(): SomeType 52 +nim fn2 subdir/subdir_b/utils.html#fn2 proc fn2() 57 +nim fn2 subdir/subdir_b/utils.html#fn2,int proc fn2(x: int) 58 +nim fn2 subdir/subdir_b/utils.html#fn2,int,float proc fn2(x: int; y: float) 61 +nim binarySearch subdir/subdir_b/utils.html#binarySearch,openArray[T],K,proc(T,K) proc binarySearch[T, K](a: openArray[T]; key: K;\n cmp: proc (x: T; y: K): int {.closure.}): int 63 +nim fn3 subdir/subdir_b/utils.html#fn3 proc fn3(): auto 66 +nim fn4 subdir/subdir_b/utils.html#fn4 proc fn4(): auto 67 +nim fn5 subdir/subdir_b/utils.html#fn5 proc fn5() 69 +nim fn6 subdir/subdir_b/utils.html#fn6 proc fn6() 70 +nim fn7 subdir/subdir_b/utils.html#fn7 proc fn7() 72 +nim fn8 subdir/subdir_b/utils.html#fn8 proc fn8(): auto 75 +nim fn9 subdir/subdir_b/utils.html#fn9,int proc fn9(a: int): int 78 +nim fn10 subdir/subdir_b/utils.html#fn10,int proc fn10(a: int): int 79 +nim fN11 subdir/subdir_b/utils.html#fN11 proc fN11() 85 +nim fN11 subdir/subdir_b/utils.html#fN11,int proc fN11(x: int) 86 +nim aEnum subdir/subdir_b/utils.html#aEnum.t template aEnum(): untyped 90 +nim bEnum subdir/subdir_b/utils.html#bEnum.t template bEnum(): untyped 95 +nim fromUtilsGen subdir/subdir_b/utils.html#fromUtilsGen.t template fromUtilsGen(): untyped 106 +nim f subdir/subdir_b/utils.html#f,G[int] proc f(x: G[int]) 130 +nim f subdir/subdir_b/utils.html#f,G[string] proc f(x: G[string]) 133 +nim `[]` subdir/subdir_b/utils.html#[],G[T] proc `[]`[T](x: G[T]): T 140 +nim `[]=` subdir/subdir_b/utils.html#[]=,G[T],int,T proc `[]=`[T](a: var G[T]; index: int; value: T) 144 +nim `$` subdir/subdir_b/utils.html#$,G[T] proc `$`[T](a: G[T]): string 148 +nim `$` subdir/subdir_b/utils.html#$,ref.SomeType proc `$`[T](a: ref SomeType): string 152 +nim fooBar subdir/subdir_b/utils.html#fooBar.i,seq[SomeType] iterator fooBar(a: seq[SomeType]): int 156 +nim fn subdir/subdir_b/utils.html#fn proc fn[T; U, V: SomeFloat]() 160 +nim `'big` subdir/subdir_b/utils.html#'big,string proc `'big`(a: string): SomeType 164 +nimgrp $ subdir/subdir_b/utils.html#$-procs-all proc 148 +nimgrp fn11 subdir/subdir_b/utils.html#fN11-procs-all proc 85 +nimgrp fn2 subdir/subdir_b/utils.html#fn2-procs-all proc 57 +nimgrp f subdir/subdir_b/utils.html#f-procs-all proc 130 +heading This is now a header subdir/subdir_b/utils.html#this-is-now-a-header This is now a header 0 +heading Next header subdir/subdir_b/utils.html#this-is-now-a-header-next-header Next header 0 +heading And so on subdir/subdir_b/utils.html#next-header-and-so-on And so on 0 +heading More headers subdir/subdir_b/utils.html#more-headers More headers 0 +heading Up to level 6 subdir/subdir_b/utils.html#more-headers-up-to-level-6 Up to level 6 0 +heading Pandoc Markdown subdir/subdir_b/utils.html#pandoc-markdown Pandoc Markdown 0 +heading Link name syntax subdir/subdir_b/utils.html#pandoc-markdown-link-name-syntax Link name syntax 0 +heading Symbols documentation subdir/subdir_b/utils.html#pandoc-markdown-symbols-documentation Symbols documentation 0 -- cgit 1.4.1-2-gfad0