Extend and document compiler debugging utilities (#19841)

* Add two debugutils procs that native debuggers can break on use to
  execute commands when code of interest is being compiled.
* Add GDB and LLDB programs to disable and enable breakpoints and
  watchpoints when code of interest is being compiled.
* Extend the `intern.rst` docs regarding debugging the compiler.

Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>

5 files changed, 280 insertions, 38 deletions

diff --git a/compiler/debugutils.nim b/compiler/debugutils.nim
index d109d2121..adbb0517f 100644
--- a/compiler/debugutils.nim
+++ b/compiler/debugutils.nim
@@ -54,3 +54,19 @@ proc isCompilerDebug*(): bool =
       {.undef(nimCompilerDebug).}
       echo 'x'
   conf0.isDefined("nimCompilerDebug")
+
+proc enteringDebugSection*() {.exportc, dynlib.} =
+  ## Provides a way for native debuggers to enable breakpoints, watchpoints, etc
+  ## when code of interest is being compiled.
+  ## 
+  ## Set your debugger to break on entering `nimCompilerIsEnteringDebugSection`
+  ## and then execute a desired command.
+  discard
+
+proc exitingDebugSection*() {.exportc, dynlib.} =
+  ## Provides a way for native debuggers to disable breakpoints, watchpoints, etc
+  ## when code of interest is no longer being compiled.
+  ## 
+  ## Set your debugger to break on entering `exitingDebugSection`
+  ## and then execute a desired command.
+  discard
diff --git a/doc/intern.rst b/doc/intern.rst
index 4ca7eff20..9103c694c 100644
--- a/doc/intern.rst
+++ b/doc/intern.rst
@@ -79,55 +79,216 @@ Set the compilation timestamp with the `SOURCE_DATE_EPOCH` environment variable.
   koch boot # or `./build_all.sh`
 
 
-Developing the compiler
-=======================
+Debugging the compiler
+======================
 
-To create a new compiler for each run, use `koch temp`:cmd:\:
+
+Bisecting for regressions
+-------------------------
+
+There are often times when there is a bug that is caused by a regression in the
+compiler or stdlib. Bisecting the Nim repo commits is a usefull tool to identify
+what commit introduced the regression.
+
+Even if it's not known whether a bug is caused by a regression, bisection can reduce
+debugging time by ruling it out. If the bug is found to be a regression, then you
+focus on the changes introduced by that one specific commit.
+
+`koch temp`:cmd: returns 125 as the exit code in case the compiler
+compilation fails. This exit code tells `git bisect`:cmd: to skip the
+current commit:
 
 .. code:: cmd
 
-  koch temp c test.nim
+  git bisect start bad-commit good-commit
+  git bisect run ./koch temp -r c test-source.nim
+
+You can also bisect using custom options to build the compiler, for example if
+you don't need a debug version of the compiler (which runs slower), you can replace
+`./koch temp`:cmd: by explicit compilation command, see `Bootstrapping the compiler`_.
+
+
+Building an instrumented compiler
+---------------------------------
+
+Considering that a useful method of debugging the compiler is inserting debug
+logging, or changing code and then observing the outcome of a testcase, it is
+fastest to build a compiler that is instrumented for debugging from an
+existing release build. `koch temp`:cmd: provides a convenient method of doing
+just that.
 
-`koch temp`:cmd: creates a debug build of the compiler, which is useful
-to create stacktraces for compiler debugging.
+By default running `koch temp`:cmd: will build a lean version of the compiler
+with `-d:debug`:option: enabled. The compiler is written to `bin/nim_temp` by
+default. A lean version of the compiler lacks JS and documentation generation.
 
-You can of course use GDB or Visual Studio to debug the
-compiler (via `--debuginfo --lineDir:on`:option:). However, there
-are also lots of procs that aid in debugging:
+`bin/nim_temp` can be directly used to run testcases, or used with testament
+with `testament --nim:bin/nim_temp r tests/category/tsometest`:cmd:.
+
+`koch temp`:cmd: will build the temporary compiler with the `-d:debug`:option:
+enabled. Here are compiler options that are of interest for debugging:
+
+* `-d:debug`:option:\: enables `assert` statements and stacktraces and all
+  runtime checks
+* `--opt:speed`:option:\: build with optimizations enabled
+* `--debugger:native`:option:\: enables `--debuginfo --lineDir:on`:option: for using
+  a native debugger like GDB, LLDB or CDB
+* `-d:nimDebug`:option: cause calls to `quit` to raise an assertion exception
+* `-d:nimDebugUtils`:option:\: enables various debugging utilities;
+  see `compiler/debugutils`
+* `-d:stacktraceMsgs -d:nimCompilerStacktraceHints`:option:\: adds some additional
+  stacktrace hints; see https://github.com/nim-lang/Nim/pull/13351
+* `-u:leanCompiler`:option:\: enable JS and doc generation
+
+Another method to build and run the compiler is directly through `koch`:cmd:\:
+
+.. code:: cmd
 
+  koch temp [options] c test.nim
+
+  # (will build with js support)
+  koch temp [options] js test.nim
+
+  # (will build with doc support)
+  koch temp [options] doc test.nim
+
+Debug logging
+-------------
+
+"Printf debugging" is still the most appropriate way to debug many problems
+arising in compiler development. The typical usage of breakpoints to debug
+the code is often less practical, because almost all of the code paths in the
+compiler will be executed hundreds of times before a particular section of the
+tested program is reached where the newly developed code must be activated.
+
+To work-around this problem, you'll typically introduce an if statement in the
+compiler code detecting more precisely the conditions where the tested feature
+is being used. One very common way to achieve this is to use the `mdbg` condition,
+which will be true only in contexts, processing expressions and statements from
+the currently compiled main module:
 
 .. code-block:: nim
 
-  # dealing with PNode:
+  # inside some compiler module
+  if mdbg:
+    debug someAstNode
+
+Using the `isCompilerDebug`:nim: condition along with inserting some statements
+into the testcase provides more granular logging:
+
+.. code-block:: nim
+
+  # compilermodule.nim
+  if isCompilerDebug():
+    debug someAstNode
+
+  # testcase.nim
+  proc main =
+    {.define(nimCompilerDebug).}
+    let a = 2.5 * 3
+    {.undef(nimCompilerDebug).}
+
+Logging can also be scoped to a specific filename as well. This will of course
+match against every module with that name.
+
+.. code-block:: nim
+
+  if `??`(conf, n.info, "module.nim"):
+    debug(n)
+
+The above examples also makes use of the `debug`:nim: proc, which is able to
+print a human-readable form of an arbitrary AST tree. Other common ways to print
+information about the internal compiler types include:
+
+.. code-block:: nim
+
+  # pretty print PNode
+
+  # pretty prints the Nim ast
   echo renderTree(someNode)
-  debug(someNode) # some JSON representation
 
-  # dealing with PType:
+  # pretty prints the Nim ast, but annotates symbol IDs
+  echo renderTree(someNode, {renderIds})
+
+  # pretty print ast as JSON
+  debug(someNode)
+
+  # print as YAML
+  echo treeToYaml(config, someNode)
+
+
+  # pretty print PType
+
+  # print type name
   echo typeToString(someType)
+
+  # pretty print as JSON
   debug(someType)
 
-  # dealing with PSym:
+  # print as YAML
+  echo typeToYaml(config, someType)
+
+
+  # pretty print PSym
+
+  # print the symbol's name
   echo symbol.name.s
+
+  # pretty print as JSON
   debug(symbol)
 
-  # pretty prints the Nim ast, but annotates symbol IDs:
-  echo renderTree(someNode, {renderIds})
-  if `??`(conf, n.info, "temp.nim"):
-    # only output when it comes from "temp.nim"
-    echo renderTree(n)
-  if `??`(conf, n.info, "temp.nim"):
-    # why does it process temp.nim here?
-    writeStackTrace()
+  # print as YAML
+  echo symToYaml(config, symbol)
+
+
+  # pretty print TLineInfo
+  lineInfoToStr(lineInfo)
+
+
+  # print the structure of any type
+  repr(someVar)
+
+Here are some other helpful utilities:
+
+.. code-block:: nim
+
+  # how did execution reach this location?
+  writeStackTrace()
 
 These procs may not already be imported by the module you're editing.
 You can import them directly for debugging:
 
 .. code-block:: nim
+
   from astalgo import debug
   from types import typeToString
   from renderer import renderTree
   from msgs import `??`
 
+Native debugging
+----------------
+
+Stepping through the compiler with a native debugger is a very powerful tool to
+both learn and debug it. However, there is still the need to constrain when
+breakpoints are triggered. The same methods as in `Debug logging`_ can be applied
+here when combined with calls to the debug helpers `enteringDebugSection()`:nim:
+and `exitingDebugSection()`:nim:.
+
+#. Compile the temp compiler with `--debugger:native -d:nimDebugUtils`:option:
+#. Set your desired breakpoints or watchpoints.
+#. Configure your debugger:
+  * GDB: execute `source tools/compiler.gdb` at startup
+  * LLDB execute `command source tools/compiler.lldb` at startup
+#. Use one of the scoping helpers like so:
+
+.. code-block:: nim
+
+  if isCompilerDebug():
+    enteringDebugSection()
+  else:
+    exitingDebugSection()
+
+A caveat of this method is that all breakpoints and watchpoints are enabled or
+disabled. Also, due to a bug, only breakpoints can be constrained for LLDB.
 
 The compiler's architecture
 ===========================
@@ -152,23 +313,6 @@ for the type definitions. The `macros <macros.html>`_ module contains many
 examples how the AST represents each syntactic structure.
 
 
-Bisecting for regressions
-=========================
-
-`koch temp`:cmd: returns 125 as the exit code in case the compiler
-compilation fails. This exit code tells `git bisect`:cmd: to skip the
-current commit:
-
-.. code:: cmd
-
-  git bisect start bad-commit good-commit
-  git bisect run ./koch temp -r c test-source.nim
-
-You can also bisect using custom options to build the compiler, for example if
-you don't need a debug version of the compiler (which runs slower), you can replace
-`./koch temp`:cmd: by explicit compilation command, see `Bootstrapping the compiler`_.
-
-
 Runtimes
 ========
 
diff --git a/doc/koch.rst b/doc/koch.rst
index b8d85dff4..91dd5d570 100644
--- a/doc/koch.rst
+++ b/doc/koch.rst
@@ -38,6 +38,9 @@ options:
   unless you are debugging the compiler.
 -d:nimUseLinenoise     Use the linenoise library for interactive mode
                        (not needed on Windows).
+-d:leanCompiler        Produce a compiler without JS codegen or
+                       documentation generator in order to use less RAM
+                       for bootstrapping.
 
 After compilation is finished you will hopefully end up with the nim
 compiler in the `bin` directory. You can add Nim's `bin` directory to
diff --git a/tools/compiler.gdb b/tools/compiler.gdb
new file mode 100644
index 000000000..c81f47152
--- /dev/null
+++ b/tools/compiler.gdb
@@ -0,0 +1,39 @@
+# create a breakpoint on `debugutils.enteringDebugSection`
+define enable_enteringDebugSection
+	break -function enteringDebugSection
+	# run these commands once breakpoint enteringDebugSection is hit
+	command
+		# enable all breakpoints and watchpoints
+		enable
+		# continue execution
+		cont
+	end
+end
+
+# create a breakpoint on `debugutils.exitingDebugSection` named exitingDebugSection
+define enable_exitingDebugSection
+	break -function exitingDebugSection
+	# run these commands once breakpoint exitingDebugSection is hit
+	command
+		# disable all breakpoints and watchpoints
+		disable
+		# but enable the enteringDebugSection breakpoint
+		enable_enteringDebugSection
+		# continue execution
+		cont
+	end
+end
+
+# some commands can't be set until the process is running, so set an entry breakpoint
+break -function NimMain
+# run these commands once breakpoint NimMain is hit
+command
+	# disable all breakpoints and watchpoints
+	disable
+	# but enable the enteringDebugSection breakpoint
+	enable_enteringDebugSection
+	# no longer need this breakpoint
+	delete -function NimMain
+	# continue execution
+	cont
+end
diff --git a/tools/compiler.lldb b/tools/compiler.lldb
new file mode 100644
index 000000000..e0b375055
--- /dev/null
+++ b/tools/compiler.lldb
@@ -0,0 +1,40 @@
+# create a breakpoint on `debugutils.enteringDebugSection` named enteringDebugSection
+breakpoint set -n 'enteringDebugSection' -N enteringDebugSection
+# run these commands once breakpoint enteringDebugSection is hit
+breakpoint command add enteringDebugSection
+	# enable all breakpoints
+	breakpoint enable
+	# enable all watchpoints
+  # watchpoint enable # FIXME: not currently working for unknown reason
+	# continue execution
+	continue
+DONE
+
+# create a breakpoint on `debugutils.exitingDebugSection` named exitingDebugSection
+breakpoint set -n 'exitingDebugSection' -N exitingDebugSection
+# run these commands once breakpoint exitingDebugSection is hit
+breakpoint command add exitingDebugSection
+	# disable all breakpoints
+	breakpoint disable
+	# disable all watchpoints
+  # watchpoint disable # FIXME: not currently working for unknown reason
+	breakpoint enable enteringDebugSection
+	# continue execution
+	continue
+DONE
+
+# some commands can't be set until the process is running, so set an entry breakpoint
+breakpoint set -n NimMain -N NimMain
+# run these commands once breakpoint NimMain is hit
+breakpoint command add NimMain
+	# disable all breakpoints
+	breakpoint disable
+	# disable all watchpoints
+  # watchpoint disable # FIXME: not currently working for unknown reason
+	# enable the enteringDebugSection breakpoint though
+	breakpoint enable enteringDebugSection
+	# no longer need this breakpoint
+	breakpoint delete NimMain
+	# continue execution
+	continue
+DONE