**Mu: making programs easier to understand in the large** Imagine a world where you can: 1. think of a tiny improvement to a program you use, clone its sources, orient yourself on its organization and make your tiny improvement, all in a single afternoon. 2. Record your program as it runs, and easily convert arbitrary logs of runs into reproducible automatic tests. 3. Answer arbitrary what-if questions about a codebase by trying out changes and seeing what tests fail, confident that *every* scenario previous authors have considered has been encoded as a test. 4. Run first simple and successively more complex versions to stage your learning. I think all these abilities might be strongly correlated; not only are they achievable with a few common concepts, but you can't easily attack one of them without also chasing after the others. The core mechanism enabling them all is recording manual tests right after the first time you perform them: * keyboard input * printing to screen * website layout * disk filling up * performance metrics * race conditions * fault tolerance * ... I hope to attain this world by creating a comprehensive library of fakes and hooks for the entire software stack, at all layers of abstraction (programming language, OS, standard libraries, application libraries). To reduce my workload and get to a proof-of-concept quickly, this is a very *alien* software stack. I've stolen ideas from lots of previous systems, but it's not like anything you're used to. The 'OS' will lack virtual memory, user accounts, any unprivileged mode, address space isolation, and many other features. To avoid building a compiler I'm going to do all my programming in (extremely type-safe) assembly (for an idealized virtual machine that nonetheless will translate easily to x86). To keep assembly from getting too painful I'm going to pervasively use one trick: load-time directives to let me order code however I want, and to write boilerplate once and insert it in multiple places. If you're familiar with literate programming or aspect-oriented programming, these directives may seem vaguely familiar. If you're not, think of them as a richer interface for function inlining. Trading off notational convenience for tests may seem regressive, but I suspect high-level languages aren't particularly helpful in understanding large codebases. No matter how good a notation is, it can only let you see a tiny fraction of a large program at a time. Logs, on the other hand, can let you zoom out and take in an entire *run* at a glance, making them a superior unit of comprehension. If I'm right, it makes sense to prioritize the right *tactile* interface for working with and getting feedback on large programs before we invest in the *visual* tools for making them concise. ([More details.](http://akkartik.name/about)) **Taking Mu for a spin** Mu is currently implemented in C++ and requires a unix-like environment. It's been tested on ubuntu 14.04 on x86, x86\_64 and ARMv7 with recent versions of gcc and clang. Since it uses no recent language features and has no exotic dependencies, it should work with most reasonable versions, compilers or processors. Running Mu will always recompile it if necessary: ```shell $ cd mu $ ./mu ``` As a sneak peek, here's how you perform some simple arithmetic: ```nim recipe example1 [ a:number <- add 2, 2 a <- multiply a, 3 ] ``` But it's easier to read in color: code example Mu functions or 'recipes' are lists of instructions, one to a line. Each instruction operates on some *ingredients* and returns some *products*. ``` [products] <- instruction [ingredients] ``` Result and ingredient *reagents* have to be variables. But you can have any number of them. In particular you can have any number of products. For example, you can perform integer division as follows: ``` quotient:number, remainder:number <- divide-with-remainder 11, 3 ``` Each reagent can provide a name as well as its type separated by a colon. You only have to specify the type the first time you mention a name, but you can be more explicit if you choose. Types can be multiple words and even arbitrary trees, like: ```nim x:array:number:3 # x is an array of 3 numbers y:list:number # y is a list of numbers # without syntactic sugar {z: (map (address array character) (list number))} # map from string to list of numbers ``` Try out the program now: ```shell $ ./mu example1.mu $ ``` Not much to see yet, since it doesn't print anything. To print the result, try adding the instruction `$print a` to the recipe. --- Here's a second example, of a recipe that can take ingredients: fahrenheit to celsius Recipes can specify headers showing their expected ingredients and products, separated by `->` (unlike the `<-` in *calls*). Since mu is a low-level VM language, it provides extra control at the cost of verbosity. Using `local-scope`, you have explicit control over stack frames to isolate your recipes (in a type-safe manner; more on that below). One consequence: you have to explicitly `load-ingredients` after you set up the stack. An alternative syntax is what the above example is converted to internally: fahrenheit to celsius desugared The header gets dropped after checking types at call-sites, and after replacing `load-ingredients` with explicit instructions to load each ingredient separately, and to explicitly return products to the caller. After this translation recipes are once again just lists of
{
  "ListenPort": 9001,
  "StdoutLogging": true,
  "LogFile": "getwtxt.log",
  "DatabasePath": "getwtxt.db",
  "DatabasePushInterval": "5m",
  "StatusFetchInterval": "1h",
  "Instance": {
    "SiteName": "getwtxt",
    "URL": "https://twtxt.example.com",
    "OwnerName": "foo barrington",
    "Email": "foo@barrington.ext",
    "Description": "This is the twtxt registry for the tildeverse network of public unix servers."
  }
}
printed to the screen -- things we don't typically associate with automated tests. b) Try skimming the [colorized source code](http://akkartik.github.io/mu). I'd like it to eventually be possible to get a pretty good sense for how things work just by skimming the files in order, skimming the top of each file and ignoring details lower down. Tell me how successful my efforts are. c) Try running the tests: ```shell $ ./mu test ``` You might also want to peek in the `.traces` directory, which automatically includes logs for each test showing you just how it ran on my machine. If Mu eventually gets complex enough that you have trouble running examples, these logs might help figure out if my system is somehow different from yours or if I've just been insufficiently diligent and my documentation is out of date. d) Try out the programming environment: ```shell $ ./mu test edit # takes about 30s; shouldn't show any failures $ ./mu edit ``` Screenshot: programming environment You write recipes on the left and try them out in *sandboxes* on the right. Hit F4 to rerun all sandboxes with the latest version of the code. More details: http://akkartik.name/post/mu. Beware, it won't save your edits by default. But if you create a sub-directory called `lesson/` under `mu/` it will. If you turn that directory into a git repo with `git init`, it will also back up each version you try out. Once you have a sandbox you can click on its result to mark it as expected: expected result Later if the result changes it'll be flagged in red to draw your attention to it. Thus, manually tested sandboxes become reproducible automated tests. unexpected result Another feature: Clicking on the code in a sandbox expands its trace for you to browse. To add to the trace, use `stash`. For example: ```nim stash [first ingredient is ], x ``` Invaluable for understanding complex control flow without cluttering up the screen. The next major milestone on Mu's roadmap is support for recording and faking console input to a sandbox, so that you can type in an input once and have it replay everytime you hit F4. Once this support is in place it will be easy to generalize to more interfaces, like requesting urls over a network or reading files on a disk. **Credits** Mu builds on many ideas that have come before, especially: - [Peter Naur](http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn,+Musashi%22) for articulating the paramount problem of programming: communicating a codebase to others; - [Christopher Alexander](http://www.amazon.com/Notes-Synthesis-Form-Harvard-Paperbacks/dp/0674627512) and [Richard Gabriel](http://dreamsongs.net/Files/PatternsOfSoftware.pdf) for the intellectual tools for reasoning about the higher order design of a codebase; - Unix and C for showing us how to co-evolve language and OS, and for teaching the (much maligned, misunderstood and underestimated) value of concise *implementation* in addition to a clean interface; - Donald Knuth's [literate programming](http://www.literateprogramming.com/knuthweb.pdf) for liberating "code for humans to read" from the tyranny of compiler order; - [David Parnas](http://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf) and others for highlighting the value of separating concerns and stepwise refinement; - [Lisp](http://www.paulgraham.com/rootsoflisp.html) for showing the power of dynamic languages, late binding and providing the right primitives *a la carte*, especially lisp macros; - The folklore of debugging by print and the trace facility in many lisp systems; - Automated tests for showing the value of developing programs inside an elaborate harness; - [Python doctest](http://docs.python.org/2/library/doctest.html) for exemplifying interactive documentation that doubles as tests; - [ReStructuredText](https://en.wikipedia.org/wiki/ReStructuredText) and [its antecedents](https://en.wikipedia.org/wiki/Setext) for showing that markup can be clean; - BDD for challenging us all to write tests at a higher level; - JavaScript and CSS for demonstrating the power of a DOM for complex structured documents.