2750 - take out all philosophy from the Readme

1 files changed, 61 insertions, 197 deletions

diff --git a/Readme.md b/Readme.md
index 29b577a1..0957dcd4 100644
--- a/Readme.md
+++ b/Readme.md
@@ -1,200 +1,64 @@
-**Mu: making programs easier to understand in the large**
-
-Mu explores an unconventional hypothesis: that all the ills of software
-directly or indirectly stem from a single historical accident -- automatic
-tests became mainstream *after* operating systems and high-level languages. As
-a result the foundations of modern software have fatal flaws that very smart
-people have tried heroically and unsuccessfully to fix at higher levels. Mu
-attempts to undo this historical accident by recreating a software stack from
-the ground up with testable interfaces. Unlike prior attempts, the goal isn't
-to create a perfectly designed programming language and operating system (I'm
-not smart enough). Rather, the goal is to preserve optionality, to avoid ever
-committing to a design decision, so that any new lessons yielded by experience
-are always able to rework all aspects of the design. The goal is to learn to
-make software *rewrite-friendly* by giving up on backwards compatibility.
-
-*The fundamental problem of software*
-
-We programmers love to chase after silver bullets, to argue the pros and cons
-of different programming languages and tools and methodologies, of whether
-rewrites are a good idea or not. When I see smart and reasonable people
-disagreeing on these questions I often track their difference down to
-variations in personal experience. In particular, people with good experiences
-of X seem disproportionately to have tried to use X at a smaller scale or
-earlier in a project's life than people with bad experiences with X. I surmise
-that this difference explains the lion's share of the benefits and drawbacks
-people observe. It doesn't matter what programming language you use, whether
-you program functionally or not, whether you follow an Object-Oriented
-methodology or go Agile, whether you use shared memory or the Actor model.
-What matters is whether you did this when the project was relatively early in
-its life; how many man-hours had been spent on it already before your
-alteration; how many people had contributed and then lost interest and moved
-on, taking some hard-won unique knowledge of the system and its users out the
-door with them. All projects decay over time and get slower to change with
-age, and it's not because of some unavoidable increase in entropy. It's
-because they grow monotonically more complex over time, because they are
-gradually boxed in by compatibility guarantees, and because their increased
-complexity makes them harder and harder for new team members to understand,
-team members who inevitably take over when the original authors inevitably
-move on.
-
-If I'm right about all this, one guiding principle shines above all others:
-keep projects easy for outsiders to navigate. It should be orders of magnitude
-easier than it is today for a reasonably competent programmer to get your code
-up and running, identify the start of the program, figure out what the major
-sub-systems are and where they connect up, run parts of your program and
-observe them in action in different situations with different inputs. All this
-should require zero hand-holding by another human, and it should require very
-little effort spent tracing through program logic. We all have the ability to
-laboriously think through what a function does, but none of us is motivated to
-do this for some strange program we've just encountered. And encountering a
-strange program is the first step for someone on the long road to becoming a
-regular contributor to your project. Make things dead simple for them. If they
-make a change, make it dead simple for them to see if it breaks something.
-
-(All this takes extra effort, and it isn't worth doing for throwaway
-prototypes, but the effort pays dividends in situations where unit tests and
-other best practices pay dividends.)
-
-But this is a hard property for a codebase to start out with, even harder to
-preserve, and impossible to regain once lost. It is a truth universally
-acknowledged that the lucid clarity of initial versions of a program is
-quickly and inevitably lost. The culprit here is monotonically growing
-complexity that makes it impossible to tell when some aspect of the program
-grows irrelevant and can be taken out. If you can't easily take something out,
-you'll never do so because there'll always be more urgent things you could be
-doing.
-
-A big source of complexity creep is a project's interface to external users,
-because you can't know all the ways in which users use a service. Historically
-we react to this constraint by assuming that our users can do anything that we
-ever allowed them to do in the past, and require ourselves to support all such
-features. We can only add features, not drop them or change how we provide
-them. We might, if we're forward thinking, keep our interfaces unstable for a
-time. But the goal is usually to stabilize the interface, and inevitably the
-stabilization is *premature*, because you can't anticipate what the future
-holds. Stable interfaces inevitably get bolted-on features, grow complex and
-breed a new generation of unsatisfied users who then attempt to wrap them in
-sane interfaces, freeze *those* interfaces, start bolting on features to them,
-rise and repeat. This dynamic of interfaces wrapping other interfaces is how
-unix cat grows from a screenful of code in the original unix to
-<a href='http://landley.net/aboriginal/history.html'>800 lines</a> in 2002 to
-36 *thousand* lines of code in 2013.
-
-To summarize, the arc you want to avoid is: you make backwards compatibility
-guarantees &rarr; complexity creeps monotonically upward &rarr; funnel of
-newcomer contributions slows &rarr; conversion of newcomers to insiders stalls
-&rarr; knowledge of the system and its rationales gradually evaporates &rarr;
-rate of change slows.
-
-But what should we replace this arc with? It's hard to imagine how a world
-could work without the division of labor that necessitates compatibility
-guarantees. Here's my tentative proposal: when you build libraries or
-interfaces for programmers, tell your users to write tests. Tell your users
-that you reserve the right to change your interface in arbitrarily subtle
-ways, and that in spite of any notifications over mailing lists and so on, the
-only situation in which it will be safe to upgrade your library is if they've
-written enough tests to cover all the edge cases of their domain that they
-care about. "It doesn't exist if you haven't written a test for it."
-
-But that's hard today because nobody's tests are that good. You can tell that
-this is true, because even the best projects with tons of tests routinely
-require manual QA when releasing a new version. A newcomer who's just starting
-to hack on your project can't do that manual QA, so he doesn't know if this
-line of code in your program is written *just so* because of some arcane
-reason of performance or just because that was the first phrasing that came to
-mind. The nice experience for an outsider would be to just change that line
-and see if any tests fail. This is only possible if we eliminate all manual QA
-from our release process.
-
-*Therefore&hellip;*
-
-In Mu, it will be possible for any aspect of any program that you can manually
-test to also be turned into a reproducible automatic test. This may seem like
-a tall order, and it is when you try to do it in a high-level language or <a
-href='http://www.seleniumhq.org'>on top of a web browser</a>. If you drop down
-to the lowest levels of your system's software, however, you find that it
-really only interacts with the outside world over a handful of modalities. The
-screen, the keyboard, the mouse, the disk, the network, maybe a couple more
-that I haven't thought of yet. All Mu has to do is make these interfaces to
-the outside world testable, give us the ability to record what we receive
-through them and replay our recordings in tests.
-
-Imagine a world where you can:
-
-1. think of a tiny improvement to a program you use, clone its sources,
-orient yourself on its organization and make your tiny improvement, all in a
-single afternoon.
-
-2. Record your program as it runs, and easily convert arbitrary logs of runs
-into reproducible automatic tests.
-
-3. Answer arbitrary what-if questions about a codebase by trying out changes
-and seeing what tests fail, confident that *every* scenario previous authors
-have considered has been encoded as a test.
-
-4. Run first simple and successively more complex versions to stage your
-learning.
-
-I think all these abilities might be strongly correlated; the right testable
-OS interfaces make them all achieveable. What's more, I can't see any way to
-attain some of these abilities without the others.
-
-As a concrete example, Unix lets you open a file by calling `open()` and
-giving it a filename. But it implicitly modifies the file system, which means
-that you can't easily call it from a test. In Mu, the `open()` syscall would
-take a file system object as an explicit argument. You'd then be able to
-access the real file system or fake it out inside a test. I'm adding similar
-explicit arguments to handle the keyboard, the network, and so on. (This
-process is called *dependency injection* and considered good practice in
-modern application software. Why shouldn't our system software evolve to
-benefit from it as well?)
-
-**Brass tacks**
-
-As you might surmise, this is a lot of work. To reduce my workload and get to
-a proof-of-concept quickly, this is a very *alien* software stack. I've stolen
-ideas from lots of previous systems, but it's not like anything you're used
-to. The 'OS' will lack virtual memory, user accounts, any unprivileged mode,
-address space isolation, and many other features.
-
-To avoid building a compiler I'm going to do all my programming in (extremely
-type-safe) assembly (for an idealized virtual machine that nonetheless is
-designed to translate easily to real processors). To keep assembly from
-getting too painful I'm going to pervasively use one trick: load-time
-directives to let me order code however I want, and to write boilerplate once
-and insert it in multiple places. If you're familiar with literate programming
-or aspect-oriented programming, these directives may seem vaguely familiar. If
-you're not, think of them as a richer interface for function inlining. (More
-details: http://akkartik.name/post/wart-layers)
-
-It probably makes you sad to have to give up the notational convenience of
-modern high-level languages. I had to go through the five stages of grief
-myself. But the benefits of the right foundation were eventually too
-compelling to resist. If I turn out to be on the right track Mu will
-eventually get high-level languages and more familiar mechanisms across the
-board. And in the meantime, I'm actually seeing signs that syntax doesn't
-really matter all that much when the goal is to understand global structure. A
-recent, more speculative hypothesis of mine is that syntax is useful for
-people who already understand the global structure of a program and who need
-to repetitively perform tasks. But if you are a newcomer to a project and you
-have a tiny peephole into it (your screen), no amount of syntactic compression
-is going to get the big picture on your screen all at once. Instead you have
-to pan around and reconstruct the big picture laboriously in your head. Tests
-help, as I've described above. Another thing that helps is a zoomable
-interface to the *trace* of operations performed in the course of a test (More
-details: http://akkartik.name/post/tracing-tests)
-
-High-level languages provide three things:
-
-a) Expressiveness (nested expressions, function calls, etc.)
-
-b) Safety (type checking, warnings, etc.)
-
-c) Automation (garbage collection, a scheduler for green threads, etc.)
-
-Mu ignores a) for a time, but there's no reason it can't provide b) and c), as
-you'll see below.
+Mu explores ways to turn arbitrary manual tests into reproducible automated
+tests. Hoped-for benefits:
+
+1. Projects release with confidence without requiring manual QA or causing
+   regressions for their users.
+
+1. Open source projects become easier for outsiders to comprehend, since they
+   can more confidently try out changes with the knowledge that they'll get
+   rapid feedback if they break something.
+
+1. It becomes easier to teach programming by emphasizing tests far earlier
+   than we do today.
+
+In this quest, Mu is currently experimenting with the following mechanisms:
+
+1. New, testable interfaces for the operating system. For example, printing to
+   screen explicitly takes a screen object, so it can be called on the real
+   screen, or on a fake inside tests, so that we can then check the expected
+   state of the screen at the end of a test. We're building up similarly
+   *dependency-injected* interfaces to the keyboard, mouse, touch screen,
+   disk, network, &hellip;
+
+1. Support for testing side-effects like performance, deadlock-freedom,
+   race-freeness, memory usage, etc. Mu's *white-box tests* can check not just
+   the results of a function call, but also the presence or absence of
+   specific events in the log of its progress. For example, if a sort function
+   logs each swap, a performance test can ensure that the number of swaps
+   doesn't double when the size of the input doubles. Besides expanding the
+   scope of tests, this ability also allows more radical refactoring without
+   needing to modify tests. All Mu's tests call a top-level function rather
+   than individual sub-systems directly. As a result the way the subsystems
+   are invoked can be radically changed (interface changes, making synchronous
+   functions asynchronous, &hellip;). As long as the new versions emit the
+   same implementation-independent events in the logs, the tests will continue
+   to pass.
+
+1. Organizing code and tests in layers of functionality, so that outsiders can
+   build simple and successively more complex versions of a project, gradually
+   enabling more peripheral features. Think of it as a cleaned-up `git log`
+   for the project. ([More information.](http://akkartik.name/post/wart-layers))
+
+Since I don't understand how Linux and other modern platforms work, Mu is
+built on an idealized VM while I learn. Eventually the plan is to transplant
+what I learn back to Linux.
+
+To minimize my workload, Mu doesn't have a high-level language yet. Instead,
+I've been programming directly in the VM's idealized assembly language. I
+expected this to be painful, but it's had some surprising benefits. First,
+programs as lists of instructions seem to be easier for non-programmers to
+comprehend than programs as trees of expressions. Second, I've found that
+Literate Programming using layers makes assembly much more ergonomic. Third,
+labels for gotos turn out to be great waypoints to insert code at from future
+layers; when I tried to divide C programs into layers, I sometimes had to
+split statements in two so I could insert code between them. They also seem a
+promising representation for providing advanced mechanisms like continuations
+and lisp-like macros.
+
+High level languages today seem to provide three kinds of benefits:
+expressiveness (e.g. nested expressions, classes), safety (e.g. type checking)
+and automation (e.g. garbage collection). An idealized assembly gives up some
+expressiveness benefits, but doesn't seem to affect the other categories.
 
 *Taking Mu for a spin*