7842 - new directory organization

Baremetal is now the default build target and therefore has its sources
at the top-level. Baremetal programs build using the phase-2 Mu toolchain
that requires a Linux kernel. This phase-2 codebase which used to be at
the top-level is now under the linux/ directory. Finally, the phase-2 toolchain,
while self-hosting, has a way to bootstrap from a C implementation, which
is now stored in linux/bootstrap. The bootstrap C implementation uses some
literate programming tools that are now in linux/bootstrap/tools.

So the whole thing has gotten inverted. Each directory should build one
artifact and include the main sources (along with standard library). Tools
used for building it are relegated to sub-directories, even though those
tools are often useful in their own right, and have had lots of interesting
programs written using them.

A couple of things have gotten dropped in this process:
  - I had old ways to run on just a Linux kernel, or with a Soso kernel.
    No more.
  - I had some old tooling for running a single test at the cursor. I haven't
    used that lately. Maybe I'll bring it back one day.

The reorg isn't done yet. Still to do:
  - redo documentation everywhere. All the README files, all other markdown,
    particularly vocabulary.md.
  - clean up how-to-run comments at the start of programs everywhere
  - rethink what to do with the html/ directory. Do we even want to keep
    supporting it?

In spite of these shortcomings, all the scripts at the top-level, linux/
and linux/bootstrap are working. The names of the scripts also feel reasonable.
This is a good milestone to take stock at.

1 files changed, 312 insertions, 0 deletions

diff --git a/linux/tests.subx b/linux/tests.subx
new file mode 100644
index 00000000..eb8eb937
--- /dev/null
+++ b/linux/tests.subx
@@ -0,0 +1,312 @@
+# Generate code for a new function called 'run-tests' which calls in sequence
+# all functions starting with 'test-'.
+#
+# To build:
+#   $ ./bootstrap translate init.linux [01]*.subx apps/subx-params.subx apps/tests.subx  -o apps/tests
+
+== code
+#   instruction                     effective address                                                   register    displacement    immediate
+# . op          subop               mod             rm32          base        index         scale       r32
+# . 1-3 bytes   3 bits              2 bits          3 bits        3 bits      3 bits        2 bits      2 bits      0/1/2/4 bytes   0/1/2/4 bytes
+
+Entry:  # run tests if necessary, convert stdin if not
+    # . prologue
+    89/copy                         3/mod/direct    5/rm32/ebp    .           .             .           4/r32/esp   .               .                 # copy esp to ebp
+
+    # initialize heap
+    # . Heap = new-segment(Heap-size)
+    # . . push args
+    68/push  Heap/imm32
+    ff          6/subop/push        0/mod/indirect  5/rm32/.disp32            .             .           .           Heap-size/disp32                  # push *Heap-size
+    # . . call
+    e8/call  new-segment/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # initialize-trace-stream(256KB)
+    # . . push args
+    68/push  0x40000/imm32/256KB
+    # . . call
+    e8/call  initialize-trace-stream/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+
+    # - if argc > 1 and argv[1] == "test", then return run_tests()
+    # if (argc <= 1) goto run-main
+    81          7/subop/compare     1/mod/*+disp8   5/rm32/ebp    .           .             .           .           0/disp8         1/imm32           # compare *ebp
+    7e/jump-if-<=  $subx-tests-main:interactive/disp8
+    # if (!kernel-string-equal?(argv[1], "test")) goto run-main
+    # . eax = kernel-string-equal?(argv[1], "test")
+    # . . push args
+    68/push  "test"/imm32
+    ff          6/subop/push        1/mod/*+disp8   5/rm32/ebp    .           .             .           .           8/disp8         .                 # push *(ebp+8)
+    # . . call
+    e8/call  kernel-string-equal?/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # . if (eax == false) goto run-main
+    3d/compare-eax-and  0/imm32/false
+    74/jump-if-=  $subx-tests-main:interactive/disp8
+    # run-tests()
+    e8/call  run-tests/disp32
+    # syscall(exit, *Num-test-failures)
+    8b/copy                         0/mod/indirect  5/rm32/.disp32            .             .           3/r32/ebx   Num-test-failures/disp32          # copy *Num-test-failures to ebx
+    eb/jump  $subx-tests-main:end/disp8
+$subx-tests-main:interactive:
+    # - otherwise convert stdin
+    # subx-gen-run-tests(Stdin, Stdout)
+    # . . push args
+    68/push  Stdout/imm32
+    68/push  Stdin/imm32
+    # . . call
+    e8/call  subx-gen-run-tests/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # syscall(exit, 0)
+    bb/copy-to-ebx  0/imm32
+$subx-tests-main:end:
+    e8/call  syscall_exit/disp32
+
+subx-gen-run-tests:  # in: (addr buffered-file), out: (addr buffered-file)
+    # pseudocode
+    #   boolean tests-found = false
+    #   var line: (stream byte 512)
+    #   var new-code-segment-handle: (handle stream byte)
+    #   new-stream(Segment-size, 1, new-code-stream-handle)
+    #   var new-code-segment: (addr stream byte) = lookup(new-code-stream-handle)
+    #   write(new-code-segment, "\n== code\n")
+    #   write(new-code-segment, "run-tests:\n")
+    #   while true
+    #     clear-stream(line)
+    #     read-line-buffered(in, line)
+    #     if (line->write == 0) break               # end of file
+    #     var word-slice = next-word(line)
+    #     if slice-empty?(word-slice)               # empty line
+    #       continue
+    #     if is-label?(word-slice)
+    #       if slice-starts-with?(word-slice, "test-")
+    #         tests-found = true
+    #         write(new-code-segment, "  e8/call  ")
+    #         write-slice(new-code-segment, word-slice)
+    #         write(new-code-segment, "/disp32\n")
+    #     rewind-stream(line)
+    #     write-stream-data(out, line)
+    #   if tests-found
+    #     write(new-code-segment, "  c3/return\n")
+    #     write-stream-data(out, new-code-segment)
+    #   flush(out)
+    #
+    # . prologue
+    55/push-ebp
+    89/copy                         3/mod/direct    5/rm32/ebp    .           .             .           4/r32/esp   .               .                 # copy esp to ebp
+    # . save registers
+    50/push-eax
+    51/push-ecx
+    52/push-edx
+    53/push-ebx
+    56/push-esi
+    57/push-edi
+    # var line/ecx: (stream byte 512)
+    81          5/subop/subtract    3/mod/direct    4/rm32/esp    .           .             .           .           .               0x200/imm32       # subtract from esp
+    68/push  0x200/imm32/length
+    68/push  0/imm32/read
+    68/push  0/imm32/write
+    89/copy                         3/mod/direct    1/rm32/ecx    .           .             .           4/r32/esp   .               .                 # copy esp to ecx
+    # var word-slice/edx: slice
+    68/push  0/imm32/end
+    68/push  0/imm32/start
+    89/copy                         3/mod/direct    2/rm32/edx    .           .             .           4/r32/esp   .               .                 # copy esp to edx
+    # tests-found?/ebx = false
+    31/xor                          3/mod/direct    3/rm32/ebx    .           .             .           3/r32/ebx   .               .                 # clear ebx
+    # var new-code-stream-handle/esi: (handle stream byte)
+    68/push  0/imm32
+    68/push  0/imm32
+    89/copy                         3/mod/direct    6/rm32/esi    .           .             .           4/r32/esp   .               .                 # copy esp to esi
+    # new-stream(Heap, Segment-size, 1, new-code-stream-handle)
+    # . . push args
+    56/push-esi
+    68/push  1/imm32
+    ff          6/subop/push        0/mod/indirect  5/rm32/.disp32            .             .           .           Segment-size/disp32               # push *Segment-size
+    68/push  Heap/imm32
+    # . . call
+    e8/call  new-stream/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               0x10/imm32        # add to esp
+    # var new-code-stream/edi: (addr stream byte) = lookup(*new-code-stream-handle)
+    # . eax = lookup(*new-code-stream-handle)
+    # . . push args
+    ff          6/subop/push        1/mod/*+disp8   6/rm32/esi    .           .             .           .           4/disp8         .                 # push *(esi+4)
+    ff          6/subop/push        0/mod/indirect  6/rm32/esi    .           .             .           .           .               .                 # push *esi
+    # . . call
+    e8/call  lookup/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # . new-code-stream = eax
+    89/copy                         3/mod/direct    7/rm32/edi    .           .             .           0/r32/eax   .               .                 # copy eax to edi
+    # write(new-code-segment, "\n== code\n")
+    # . . push args
+    68/push  "\n== code\n"/imm32
+    57/push-edi
+    # . . call
+    e8/call  write/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # write(new-code-segment, "run-tests:\n")
+    # . . push args
+    68/push  "run-tests:\n"/imm32
+    57/push-edi
+    # . . call
+    e8/call  write/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+$subx-gen-run-tests:loop:
+    # clear-stream(line)
+    # . . push args
+    51/push-ecx
+    # . . call
+    e8/call  clear-stream/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+    # read-line-buffered(in, line)
+    # . . push args
+    51/push-ecx
+    ff          6/subop/push        1/mod/*+disp8   5/rm32/ebp    .           .             .           .           8/disp8         .                 # push *(ebp+8)
+    # . . call
+    e8/call  read-line-buffered/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+$subx-gen-run-tests:check0:
+    # if (line->write == 0) break
+    81          7/subop/compare     0/mod/indirect  1/rm32/ecx    .           .             .           .           .               0/imm32           # compare *ecx
+    0f 84/jump-if-=  $subx-gen-run-tests:break/disp32
+    # next-word(line, word-slice)
+    # . . push args
+    52/push-edx
+    51/push-ecx
+    # . . call
+    e8/call  next-word/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+$subx-gen-run-tests:check-empty:
+    # if slice-empty?(word-slice) break
+    # . eax = slice-empty?(word-slice)
+    52/push-edx
+    e8/call  slice-empty?/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+    # . if (eax != false) break
+    3d/compare-eax-and  0/imm32/false
+    75/jump-if-!=  $subx-gen-run-tests:loop/disp8
+$subx-gen-run-tests:check-for-label:
+    # if (!is-label?(word-slice)) continue
+    # . eax = is-label?(word-slice)
+    # . . push args
+    52/push-edx
+    # . . call
+    e8/call  is-label?/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+    # . if (eax == false) continue
+    3d/compare-eax-and  0/imm32/false
+    74/jump-if-=  $subx-gen-run-tests:continue/disp8
+$subx-gen-run-tests:check-label-prefix:
+    # strip trailing ':' from word-slice
+    ff          1/subop/decrement   1/mod/*+disp8   2/rm32/edx    .           .             .           .           4/disp8         .                 # decrement *(edx+4)
+    # if !slice-starts-with?(word-slice, "test-") continue
+    # . . push args
+    68/push  "test-"/imm32
+    52/push-edx
+    # . . call
+    e8/call  slice-starts-with?/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # . if (eax == false) break
+    3d/compare-eax-and  0/imm32/false
+    74/jump-if-=  $subx-gen-run-tests:continue/disp8
+$subx-gen-run-tests:call-test-function:
+    # tests-found? = true
+    bb/copy-to-ebx  1/imm32/true
+    # write(new-code-segment, "  e8/call  ")
+    # . . push args
+    68/push  "  e8/call  "/imm32
+    57/push-edi
+    # . . call
+    e8/call  write/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # write-slice(new-code-segment, word-slice)
+    # . . push args
+    52/push-edx
+    57/push-edi
+    # . . call
+    e8/call  write-slice/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # write(new-code-segment, "/disp32\n")
+    # . . push args
+    68/push  "/disp32\n"/imm32
+    57/push-edi
+    # . . call
+    e8/call  write/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+$subx-gen-run-tests:continue:
+    # rewind-stream(line)
+    # . . push args
+    51/push-ecx
+    # . . call
+    e8/call  rewind-stream/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+    # write-stream-data(out, line)
+    # . . push args
+    51/push-ecx
+    ff          6/subop/push        1/mod/*+disp8   5/rm32/ebp    .           .             .           .           0xc/disp8       .                 # push *(ebp+12)
+    # . . call
+    e8/call  write-stream-data/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # loop
+    e9/jump  $subx-gen-run-tests:loop/disp32
+$subx-gen-run-tests:break:
+    # if (!tests-found?) goto end
+    81          7/subop/compare     3/mod/direct    3/rm32/ebx    .           .             .           .           .               0/imm32/false     # compare ebx
+    74/jump-if-=  $subx-gen-run-tests:end/disp8
+    # write(new-code-segment, "  c3/return\n")
+    # . . push args
+    68/push  "  c3/return\n"/imm32
+    57/push-edi
+    # . . call
+    e8/call  write/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+    # write-stream-data(out, new-code-segment)
+    # . . push args
+    57/push-edi
+    ff          6/subop/push        1/mod/*+disp8   5/rm32/ebp    .           .             .           .           0xc/disp8       .                 # push *(ebp+12)
+    # . . call
+    e8/call  write-stream-data/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               8/imm32           # add to esp
+$subx-gen-run-tests:end:
+    # flush(out)
+    # . . push args
+    ff          6/subop/push        1/mod/*+disp8   5/rm32/ebp    .           .             .           .           0xc/disp8       .                 # push *(ebp+12)
+    # . . call
+    e8/call  flush/disp32
+    # . . discard args
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               4/imm32           # add to esp
+    # . reclaim locals
+    81          0/subop/add         3/mod/direct    4/rm32/esp    .           .             .           .           .               0x21c/imm32       # add to esp
+    # . restore registers
+    5f/pop-to-edi
+    5e/pop-to-esi
+    5b/pop-to-ebx
+    5a/pop-to-edx
+    59/pop-to-ecx
+    58/pop-to-eax
+    # . epilogue
+    89/copy                         3/mod/direct    4/rm32/esp    .           .             .           5/r32/ebp   .               .                 # copy ebp to esp
+    5d/pop-to-ebp
+    c3/return
+
+# . . vim:nowrap:textwidth=0