7843 - clean up README after directory reorg

4 files changed, 160 insertions, 15 deletions

diff --git a/linux/README.md b/linux/README.md
index 23b61058..5eabd571 100644
--- a/linux/README.md
+++ b/linux/README.md
@@ -1,7 +1,8 @@
+A set of standard libraries for building programs that run with just a Linux
+kernel. Most programs here read from stdin and write to stdout. One of these
+programs is the Mu compiler ([colorized sources](http://akkartik.github.io/mu/html/apps/mu.subx.html)).
 
-
-
-Some apps written in SubX and Mu. Check out:
+Other apps beyond the Mu toolchain:
 
 * `tile`: [An experimental live-updating postfix shell environment](https://mastodon.social/@akkartik/105108305362341204)
   that updates as you type. Prototype. Look at this to see what is currently
@@ -17,15 +18,13 @@ Some apps written in SubX and Mu. Check out:
 * `factorial*`: A simple program to compute factorials in 5 versions, showing
   all the different syntax sugars and what they expand to.
 
-* Code unique to phases of our build toolchain:
-  * Core SubX: `hex`, `survey_elf`, `pack`, `dquotes`, `assort`, `tests`
-  * Syntax sugar for SubX: `sigils`, `calls`, `braces`
-  * More ambitious translator for a memory-safe language (in progress): `mu`
-
-* Miscellaneous test programs.
+The Mu toolchain is also here in the following phases:
+* Core SubX: `hex`, `survey_elf`, `pack`, `dquotes`, `assort`, `tests`
+* Syntax sugar for SubX: `sigils`, `calls`, `braces`
+* More ambitious translator for a memory-safe language (in progress): `mu`
 
-All SubX apps include binaries. At any commit, an example's binary should be
-identical bit for bit with the result of translating the corresponding `.subx`
-file. The binary should also be natively runnable on a Linux system running on
-Intel x86 processors, either 32- or 64-bit. If either of these invariants is
-violated, it's a bug.
+The toolchain includes binaries in the repo. At any commit, the binary should
+be identical bit for bit with the result of translating the corresponding
+`.subx` file. The binary should also be natively runnable on a Linux system
+running on Intel x86 processors, either 32- or 64-bit. If either of these
+invariants is violated, it's a bug.
diff --git a/linux/bootstrap/README.md b/linux/bootstrap/README.md
index fdc3213a..ca4ea4b3 100644
--- a/linux/bootstrap/README.md
+++ b/linux/bootstrap/README.md
@@ -4,4 +4,6 @@ a) An emulator for SubX, the subset of the 32-bit x86 instruction set used by
 Mu.
 
 b) A second translator for SubX programs that emits identical binaries to the
-self-hosting versions in the parent directory.
+self-hosting versions in the parent directory. Having two diverse compilers
+(one in a familiar language, one with minimal syscall surface area) that emit
+identical binaries should help gain confidence in Mu.
diff --git a/linux/bootstrap/bootstrap.md b/linux/bootstrap/bootstrap.md
new file mode 100644
index 00000000..ca9320e5
--- /dev/null
+++ b/linux/bootstrap/bootstrap.md
@@ -0,0 +1,17 @@
+## Running
+
+`bootstrap` currently has the following sub-commands:
+
+- `bootstrap help`: some helpful documentation to have at your fingertips.
+
+- `bootstrap test`: runs all automated tests.
+
+- `bootstrap translate <input files> -o <output ELF binary>`: translates `.subx`
+  files into an executable ELF binary.
+
+- `bootstrap run <ELF binary> <args>`: simulates running the ELF binaries emitted
+  by `bootstrap translate`. Useful for testing and debugging.
+
+  Remember, not all 32-bit Linux binaries are guaranteed to run. I'm not
+  building general infrastructure here for all of the x86 instruction set.
+  SubX is about programming with a small, regular subset of 32-bit x86.
diff --git a/linux/subx_debugging.md b/linux/subx_debugging.md
new file mode 100644
index 00000000..e179df54
--- /dev/null
+++ b/linux/subx_debugging.md
@@ -0,0 +1,127 @@
+## A few hints for debugging SubX programs
+
+Writing programs in SubX is surprisingly pleasant and addictive. Reading
+programs is a work in progress, and hopefully the extensive unit tests help.
+However, _debugging_ programs is where one really faces up to the low-level
+nature of SubX. Even the smallest modifications need testing to make sure they
+work. In my experience, there is no modification so small that I get it working
+on the first attempt. And when it doesn't work, there are no clear error
+messages. Machine code is too simple-minded for that. You can't use a debugger,
+since SubX's simplistic ELF binaries contain no debugging information. So
+debugging requires returning to basics and practicing with a new, more
+rudimentary but hopefully still workable toolkit:
+
+- Start by nailing down a concrete set of steps for reproducibly obtaining the
+  error or erroneous behavior.
+
+- If possible, turn the steps into a failing test. It's not always possible,
+  but SubX's primary goal is to keep improving the variety of tests one can
+  write.
+
+- Start running the single failing test alone. This involves modifying the top
+  of the program (or the final `.subx` file passed in to `bootstrap translate`) by
+  replacing the call to `run-tests` with a call to the appropriate `test-`
+  function.
+
+- Generate a trace for the failing test while running your program in emulated
+  mode (`bootstrap run`):
+
+  ```
+  $ cd linux
+  $ ./translate_subx_debug file1.subx file2.subx ...  # generating a.elf
+  $ ./bootstrap --trace run a.elf arg1 arg2
+  saving trace to 'last_run'
+  ```
+
+  The ability to generate a trace is the essential reason for the existence of
+  `bootstrap run` mode. It gives far better visibility into program internals than
+  running natively.
+
+  Here's a sample of the contents of `last_run`, with a few boxes highlighted:
+
+  <img alt='trace example' src='html/trace.png'>
+
+  Each of the green boxes shows the trace emitted for a single instruction.
+  It starts with a line of the form `run: inst: ___` followed by the opcode
+  for the instruction, the state of registers before the instruction executes,
+  and various other facts deduced during execution. Some instructions first
+  print a matching label. In the above screenshot, the red boxes show that
+  address `0x0900005e` maps to label `$loop` and presumably marks the start of
+  some loop. Function names get similar `run: == label` lines.
+
+- One quick trick when scanning a trace for the first time:
+
+  ```
+  $ grep label last_run
+  ```
+
+  This is useful for quickly showing you the control flow for the run, and the
+  function executing when the error occurred. I find it useful to start with
+  this information, only looking at the complete trace after I've gotten
+  oriented on the control flow. Did it get to the loop I just modified? How
+  many times did it go through the loop?
+
+- Once you have SubX displaying labels in traces, it's a short step to modify
+  the program to insert more labels just to gain more insight. For example,
+  consider the following function:
+
+  <img alt='control example -- before' src='html/control0.png'>
+
+  This function contains a series of jump instructions. If a trace shows
+  `is-hex-lowercase-byte?` being encountered, and then `$is-hex-lowercase-byte?:end`
+  being encountered, it's still ambiguous what happened. Did we hit an early
+  exit, or did we execute all the way through? To clarify this, add temporary
+  labels after each jump:
+
+  <img alt='control example -- after' src='html/control1.png'>
+
+  Now the trace should have a lot more detail on which of these labels was
+  reached, and precisely when the exit was taken.
+
+- If you find yourself wondering, "when did the contents of this memory
+  address change?", `bootstrap run` has some rudimentary support for _watch
+  points_. Just insert a label starting with `$watch-` before an instruction
+  that writes to the address, and its value will start getting dumped to the
+  trace after every instruction thereafter.
+
+- Once we have a sense for precisely which instructions we want to look at,
+  it's time to look at the trace as a whole. Key is the state of registers
+  before each instruction. If a function is receiving bad arguments it becomes
+  natural to inspect what values were pushed on the stack before calling it,
+  tracing back further from there, and so on.
+
+  I occasionally want to see the precise state of the stack segment, in which
+  case I uncomment a commented-out call to `dump_stack()` in the `vm.cc`
+  layer. It makes the trace a lot more verbose and a lot less dense, necessitating
+  a lot more scrolling around, so I keep it turned off most of the time.
+
+- If the trace seems overwhelming, try [browsing it](https://github.com/akkartik/mu/blob/master/tools/browse_trace.readme.md)
+  in the 'time-travel debugger'.
+
+- Don't be afraid to slice and dice the trace using Unix tools. For example,
+  say you have a SubX binary that dies while running tests. You can see what
+  test it's segfaulting at by compiling it with debug information using
+  `./translate_subx_debug`, and then running:
+
+  ```
+  grep 'label test-' |tail
+  ```
+
+  Just read out the last test printed out before the segfault.
+
+  Even outside of tests, I can often quickly debug an error just by scanning
+  the end of a trace for labels:
+
+  ```
+  $ grep label last_run |tail
+  ```
+
+  Knowing _where_ the error occurred is often enough to put me on the right
+  track to debugging an error.
+
+Hopefully these hints are enough to get you started. The main thing to
+remember is to not be afraid of modifying the sources. A good debugging
+session gets into a nice rhythm of generating a trace, staring at it for a
+while, modifying the sources, regenerating the trace, and so on. Email
+[me](mailto:mu@akkartik.com) if you'd like another pair of eyes to stare at a
+trace, or if you have questions or complaints.