; Mu: An exploration on making the global structure of programs more accessible. ; ; "Is it a language, or an operating system, or a virtual machine? Mu." ; (with apologies to Robert Pirsig: http://en.wikipedia.org/wiki/Mu_%28negative%29#In_popular_culture) ; ; I want to live in a world where I can have an itch to tweak a program, clone ; its open-source repository, orient myself on how it's organized, and make ; the simple change I envisioned, all in an afternoon. This codebase tries to ; make this possible for its readers. (More details: http://akkartik.name/about) ; ; What helps comprehend the global structure of programs? For starters, let's ; enumerate what doesn't: idiomatic code, adherence to a style guide or naming ; convention, consistent indentation, API documentation for each class, etc. ; These conventional considerations improve matters in the small, but don't ; help understand global organization. They help existing programmers manage ; day-to-day operations, but they can't turn outsider programmers into ; insiders. (Elaboration: http://akkartik.name/post/readable-bad) ; ; In my experience, two things have improved matters so far: version control ; and automated tests. Version control lets me rewind back to earlier, simpler ; times when the codebase was simpler, when its core skeleton was easier to ; ascertain. Indeed, arguably what came first is by definition the skeleton of ; a program, modulo major rewrites. Once you understand the skeleton, it ; becomes tractable to 'play back' later major features one by one. (Previous ; project that fleshed out this idea: http://akkartik.name/post/wart-layers) ; ; The second and biggest boost to comprehension comes from tests. Tests are ; good for writers for well-understood reasons: they avoid regressions, and ; they can influence code to be more decoupled and easier to change. In ; addition, tests are also good for the outsider reader because they permit ; active reading. If you can't build a program and run its tests it can't help ; you understand it. It hangs limp at best, and might even be actively ; misleading. If you can run its tests, however, it comes alive. You can step ; through scenarios in a debugger. You can add logging and scan logs to make ; sense of them. You can run what-if scenarios: "why is this line not written ; like this?" Make a change, rerun tests: "Oh, that's why." (Elaboration: ; http://akkartik.name/post/literate-programming) ; ; However, tests are only useful to the extent that they exist. Think back to ; your most recent codebase. Do you feel comfortable releasing a new version ; just because the tests pass? I'm not aware of any such project. There's just ; too many situations envisaged by the authors that were never encoded in a ; test. Even disciplined authors can't test for performance or race conditions ; or fault tolerance. If a line is phrased just so because of some subtle ; performance consideration, it's hard to communicate to newcomers. ; ; This isn't an arcane problem, and it isn't just a matter of altruism. As ; more and more such implicit considerations proliferate, and as the original ; authors are replaced by latecomers for day-to-day operations, knowledge is ; actively forgotten and lost. The once-pristine codebase turns into legacy ; code that is hard to modify without expensive and stress-inducing ; regressions. ; ; How to write tests for performance, fault tolerance, race conditions, etc.? ; How can we state and verify that a codepath doesn't ever perform memory ; allocation, or write to disk? It requires better, more observable primitives ; than we currently have. Modern operating systems have their roots in the ; 70s. Their interfaces were not designed to be testable. They provide no way ; to simulate a full disk, or a specific sequence of writes from different ; threads. We need something better. ; ; This project tries to move, groping, towards that 'something better', a ; platform that is both thoroughly tested and allows programs written for it ; to be thoroughly tested. It tries to answer the question: ; ; If Denis Ritchie and Ken Thompson were to set out today to co-design unix ; and C, knowing what we know about automated tests, what would they do ; differently? ; ; To try to impose *some* constraints on this gigantic yak-shave, we'll try to ; keep both language and OS as simple as possible, focused entirely on ; permitting more kinds of tests, on first *collecting* all the information ; about implicit considerations in some form so that readers and tools can ; have at least some hope of making sense of it. ; ; The initial language will be just assembly. We'll try to make it convenient ; to program in with some simple localized rewrite rules inspired by lisp ; macros and literate programming. Programmers will have to do their own ; memory management and register allocation, but we'll provide libraries to ; help with them. ; ; The initial OS will provide just memory management and concurrency ; primitives. No users or permissions (we don't live on mainframes anymore), ; no kernel- vs user-mode, no virtual memory or process abstraction, all ; threads sharing a single address space (use VMs for security and ; sandboxing). The only use case we care about is getting a test harness to ; run some code, feed it data through blocking channels, stop it and observe ; its internals. The code under test is expected to cooperate in such testing, ; by logging important events for the test harness to observe. (More info: ; http://akkartik.name/post/tracing-tests) ; ; The common thread here is elimination of abstractions, and it's not an ; accident. Abstractions help insiders manage the evolution of a codebase, but ; they actively hinder outsiders in understanding it from scratch. This ; matters, because the funnel to turn outsiders into insiders is critical to ; the long-term life of a codebase. Perhaps authors should raise their ; estimation of the costs of abstraction, and go against their instincts for ; introducing it. That's what I'll be trying to do: question every abstraction ; before I introduce it. We'll see how it goes. ; --- ; Mu is currently built atop Racket and Arc, but this is temporary and ; contingent. We want to keep our options open, whether to port to a different ; host language, and easy to rewrite to native code for any platform. So we'll ; try to avoid 'cheating': relying on the host platform for advanced ; functionality. ; ; Other than that, we'll say no more about the code, and focus in the rest of ; this file on the scenarios the code cares about. (load "mu.arc") ; Our language is assembly-like in that functions consist of series of ; statements, and statements consist of an operation and its arguments (input ; and output). ; ; oarg1, oarg2, ... <- op arg1, arg2, ... ; ; Args must be atomic, like an integer or a memory address, they can't be ; expressions doing arithmetic or function calls. But we can have any number ; of them. ; ; Since we're building on lisp, our code samples won't look quite like the ; idealized syntax above. For now they will be lists of lists: ; ; (function-name ; ((oarg1 oarg2 ... <- op arg1 arg2 ...) ; ... ; ...)) ; ; Each arg/oarg is itself a list, with the payload value at the head, and ; various metadata in the rest. In this first example the only metadata is types: ; 'integer' for a memory location containing an integer, and 'literal' for a ; value included directly in code. (Assembly languages traditionally call them ; 'immediate' operands.) In the future a simple tool will check that the types ; line up as expected in each op. A different tool might add types where they ; aren't provided. Instead of a monolithic compiler I want to build simple, ; lightweight tools that can be combined in various ways, say for using ; different typecheckers in different subsystems. (reset) (new-trace "literal") (add-fns '((main ((1 integer) <- copy (23 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 23) (prn "F - 'copy' writes its lone 'arg' after the instruction name to its lone 'oarg' or output arg before the arrow. After this test, the value 23 is stored in memory address 1.")) ;? (quit) ; Our basic arithmetic ops can operate on memory locations or literals. ; (Ignore hardware details like registers for now.) (reset) (new-trace "add") (add-fns '((main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) ((3 integer) <- add (1 integer) (2 integer))))) (run 'main) (if (~iso memory* (obj 1 1 2 3 3 4)) (prn "F - 'add' operates on two addresses")) (reset) (new-trace "add-literal") (add-fns '((main ((1 integer) <- add (2 literal) (3 literal))))) (run 'main) (if (~is memory*.1 5) (prn "F - ops can take 'literal' operands (but not return them)")) (reset) (new-trace "sub-literal") (add-fns '((main ((1 integer) <- sub (1 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 -2) (prn "F - 'sub' subtracts the second arg from the first")) (reset) (new-trace "mul-literal") (add-fns '((main ((1 integer) <- mul (2 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 6) (prn "F - 'mul' multiplies like 'add' adds")) (reset) (new-trace "div-literal") (add-fns '((main ((1 integer) <- div (8 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 (/ real.8 3)) (prn "F - 'div' divides like 'sub' subtracts")) (reset) (new-trace "idiv-literal") (add-fns '((main ((1 integer) (2 integer) <- idiv (8 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 2)) (prn "F - 'idiv' performs integer division, returning quotient and remainder")) ; Basic boolean operations: and, or, not ; There are easy ways to encode booleans in binary, but we'll skip past those ; details. (reset) (new-trace "and-literal") (add-fns '((main ((1 boolean) <- and (t literal) (nil literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 nil) (prn "F - logical 'and' for booleans")) ; Basic comparison operations: lt, le, gt, ge, eq, neq (reset) (new-trace "lt-literal") (add-fns '((main ((1 boolean) <- lt (4 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 nil) (prn "F - 'lt' is the less-than inequality operator")) (reset) (new-trace "le-literal-false") (add-fns '((main ((1 boolean) <- le (4 literal) (3 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 nil) (prn "F - 'le' is the <= inequality operator")) (reset) (new-trace "le-literal-true") (add-fns '((main ((1 boolean) <- le (4 literal) (4 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 t) (prn "F - 'le' returns true for equal operands")) (reset) (new-trace "le-literal-true-2") (add-fns '((main ((1 boolean) <- le (4 literal) (5 literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 t) (prn "F - le is the <= inequality operator - 2")) ; Control flow operations: jump, jump-if ; These introduce a new type -- 'offset' -- for literals that refer to memory ; locations relative to the current location. (reset) (new-trace "jump-skip") (add-fns '((main ((1 integer) <- copy (8 literal)) (jump (1 offset)) ((2 integer) <- copy (3 literal)) ; should be skipped (reply)))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 8)) (prn "F - 'jump' skips some instructions")) (reset) (new-trace "jump-target") (add-fns '((main ((1 integer) <- copy (8 literal)) (jump (1 offset)) ((2 integer) <- copy (3 literal)) ; should be skipped (reply) ((3 integer) <- copy (34 literal))))) ; never reached (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 8)) (prn "F - 'jump' doesn't skip too many instructions")) ;? (quit) (reset) (new-trace "jump-if-skip") (add-fns '((main ((2 integer) <- copy (1 literal)) ((1 boolean) <- eq (1 literal) (2 integer)) (jump-if (1 boolean) (1 offset)) ((2 integer) <- copy (3 literal)) (reply) ((3 integer) <- copy (34 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 t 2 1)) (prn "F - 'jump-if' is a conditional 'jump'")) (reset) (new-trace "jump-if-fallthrough") (add-fns '((main ((1 boolean) <- eq (1 literal) (2 literal)) (jump-if (3 boolean) (1 offset)) ((2 integer) <- copy (3 literal)) (reply) ((3 integer) <- copy (34 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 nil 2 3)) (prn "F - if 'jump-if's first arg is false, it doesn't skip any instructions")) (reset) (new-trace "jump-if-backward") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (1 literal)) ; loop ((2 integer) <- add (2 integer) (2 integer)) ((3 boolean) <- eq (1 integer) (2 integer)) (jump-if (3 boolean) (-3 offset)) ; to loop ((4 integer) <- copy (3 literal)) (reply) ((3 integer) <- copy (34 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 4 3 nil 4 3)) (prn "F - 'jump-if' can take a negative offset to make backward jumps")) ; Data movement relies on addressing modes: ; 'direct' - refers to a memory location; default for most types. ; 'literal' - directly encoded in the code; implicit for some types like 'offset'. (reset) (new-trace "direct-addressing") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 integer) <- copy (1 integer))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 34)) (prn "F - 'copy' performs direct addressing")) ; 'Indirect' addressing refers to an address stored in a memory location. ; Indicated by the metadata 'deref'. Usually requires an address type. ; In the test below, the memory location 1 contains '2', so an indirect read ; of location 1 returns the value of location 2. (reset) (new-trace "indirect-addressing") (add-fns '((main ((1 integer-address) <- copy (2 literal)) ((2 integer) <- copy (34 literal)) ((3 integer) <- copy (1 integer-address deref))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 34 3 34)) (prn "F - 'copy' performs indirect addressing")) ; Output args can use indirect addressing. In the test below the value is ; stored at the location stored in location 1 (i.e. location 2). (reset) (new-trace "indirect-addressing-oarg") (add-fns '((main ((1 integer-address) <- copy (2 literal)) ((2 integer) <- copy (34 literal)) ((1 integer-address deref) <- add (2 integer) (2 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 36)) (prn "F - instructions can perform indirect addressing on output arg")) ; Until now we've dealt with scalar types like integers and booleans and ; addresses, where mu looks like other assembly languages. In addition, mu ; provides first-class support for compound types: arrays and records. ; ; 'get' accesses fields in records ; 'index' accesses indices in arrays ; ; Both operations require knowledge about the types being worked on, so all ; types used in mu programs are defined in a single global system-wide table ; (see types* in mu.arc for the complete list of types; we'll add to it over ; time). (reset) (new-trace "get-record") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (nil literal)) ((3 boolean) <- get (1 integer-boolean-pair) (1 offset)) ((4 integer) <- get (1 integer-boolean-pair) (0 offset))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 nil 3 nil 4 34)) (prn "F - 'get' accesses fields of records")) (reset) (new-trace "get-indirect") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (nil literal)) ((3 integer-boolean-pair-address) <- copy (1 literal)) ((4 boolean) <- get (3 integer-boolean-pair-address deref) (1 offset)) ((5 integer) <- get (3 integer-boolean-pair-address deref) (0 offset))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 nil 3 1 4 nil 5 34)) (prn "F - 'get' accesses fields of record address")) (reset) (new-trace "get-compound-field") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 integer) <- copy (35 literal)) ((3 integer) <- copy (36 literal)) ((4 integer-integer-pair) <- get (1 integer-point-pair) (1 offset))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 35 3 36 4 35 5 36)) (prn "F - 'get' accesses fields spanning multiple locations")) (reset) (new-trace "get-address") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (t literal)) ((3 boolean-address) <- get-address (1 integer-boolean-pair) (1 offset))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 t 3 2)) (prn "F - 'get-address' returns address of fields of records")) (reset) (new-trace "get-address-indirect") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (t literal)) ((3 integer-boolean-pair-address) <- copy (1 literal)) ((4 boolean-address) <- get-address (3 integer-boolean-pair-address deref) (1 offset))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 t 3 1 4 2)) (prn "F - 'get-address' accesses fields of record address")) (reset) (new-trace "index-array-literal") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (23 literal)) ((3 boolean) <- copy (nil literal)) ((4 integer) <- copy (24 literal)) ((5 boolean) <- copy (t literal)) ((6 integer-boolean-pair) <- index (1 integer-boolean-pair-array) (1 literal))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 23 3 nil 4 24 5 t 6 24 7 t)) (prn "F - 'index' accesses indices of arrays")) (reset) (new-trace "index-array-direct") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (23 literal)) ((3 boolean) <- copy (nil literal)) ((4 integer) <- copy (24 literal)) ((5 boolean) <- copy (t literal)) ((6 integer) <- copy (1 literal)) ((7 integer-boolean-pair) <- index (1 integer-boolean-pair-array) (6 integer))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 23 3 nil 4 24 5 t 6 1 7 24 8 t)) (prn "F - 'index' accesses indices of arrays")) (reset) (new-trace "index-address") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (23 literal)) ((3 boolean) <- copy (nil literal)) ((4 integer) <- copy (24 literal)) ((5 boolean) <- copy (t literal)) ((6 integer) <- copy (1 literal)) ((7 integer-boolean-pair-address) <- index-address (1 integer-boolean-pair-array) (6 integer))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 23 3 nil 4 24 5 t 6 1 7 4)) (prn "F - 'index-address' returns addresses of indices of arrays")) ; Array values know their length. Record lengths are saved in the types table. (reset) (new-trace "len-array") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (23 literal)) ((3 boolean) <- copy (nil literal)) ((4 integer) <- copy (24 literal)) ((5 boolean) <- copy (t literal)) ((6 integer) <- len (1 integer-boolean-pair-array))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 2 2 23 3 nil 4 24 5 t 6 2)) (prn "F - 'len' accesses length of array")) ; 'sizeof' is a helper to determine the amount of memory required by a type. (reset) (new-trace "sizeof-record") (add-fns '((main ((1 integer) <- sizeof (integer-boolean-pair literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 2) (prn "F - 'sizeof' returns space required by arg")) (reset) (new-trace "sizeof-record-not-len") (add-fns '((main ((1 integer) <- sizeof (integer-point-pair literal))))) (run 'main) ;? (prn memory*) (if (~is memory*.1 3) (prn "F - 'sizeof' is different from number of elems")) ; Regardless of a type's length, you can move it around just like a primitive. (reset) (new-trace "compound-operand-copy") (add-fns '((main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (nil literal)) ((4 boolean) <- copy (t literal)) ((3 integer-boolean-pair) <- copy (1 integer-boolean-pair))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 nil 3 34 4 nil)) (prn "F - ops can operate on records spanning multiple locations")) (reset) (new-trace "compound-arg") (add-fns '((test1 ((4 integer-boolean-pair) <- arg)) (main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (nil literal)) (test1 (1 integer-boolean-pair))))) (run 'main) (if (~iso memory* (obj 1 34 2 nil 4 34 5 nil)) (prn "F - 'arg' can copy records spanning multiple locations")) (reset) (new-trace "compound-arg") ;? (set dump-trace*) (add-fns '((test1 ((4 integer-boolean-pair) <- arg)) (main ((1 integer) <- copy (34 literal)) ((2 boolean) <- copy (nil literal)) ((3 integer-boolean-pair-address) <- copy (1 literal)) (test1 (3 integer-boolean-pair-address deref))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 nil 3 1 4 34 5 nil)) (prn "F - 'arg' can copy records spanning multiple locations in indirect mode")) ; A special kind of record is the 'tagged type'. It lets us represent ; dynamically typed values, which save type information in memory rather than ; in the code to use them. This will let us do things like create heterogenous ; lists containing both integers and strings. Tagged values admit two ; operations: ; ; 'save-type' - turns a regular value into a tagged-value of the appropriate type ; 'maybe-coerce' - turns a tagged value into a regular value if the type matches (reset) (new-trace "tagged-value") ;? (set dump-trace*) (add-fns '((main ((1 type) <- copy (integer-address literal)) ((2 integer-address) <- copy (34 literal)) ; pointer to nowhere ((3 integer-address) (4 boolean) <- maybe-coerce (1 tagged-value) (integer-address literal))))) (run 'main) ;? (prn memory*) (if (or (~is memory*.3 34) (~is memory*.4 t)) (prn "F - 'maybe-coerce' copies value only if type tag matches")) (reset) (new-trace "tagged-value-2") ;? (set dump-trace*) (add-fns '((main ((1 type) <- copy (integer-address literal)) ((2 integer-address) <- copy (34 literal)) ; pointer to nowhere ((3 integer-address) (4 boolean) <- maybe-coerce (1 tagged-value) (boolean-address literal))))) (run 'main) ;? (prn memory*) (if (or (~is memory*.3 0) (~is memory*.4 nil)) (prn "F - 'maybe-coerce' doesn't copy value when type tag doesn't match")) (reset) (new-trace "save-type") (add-fns '((main ((1 integer-address) <- copy (34 literal)) ; pointer to nowhere ((2 tagged-value) <- save-type (1 integer-address))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 34 2 'integer-address 3 34)) (prn "F - 'save-type' saves the type of a value at runtime, turning it into a tagged-value")) (reset) (new-trace "new-tagged-value") ;? (set dump-trace*) (add-fns '((main ((1 integer-address) <- copy (34 literal)) ; pointer to nowhere ((2 tagged-value-address) <- new-tagged-value (integer-address literal) (1 integer-address)) ((3 integer-address) (4 boolean) <- maybe-coerce (2 tagged-value-address deref) (integer-address literal))))) (run 'main) ;? (prn memory*) (if (or (~is memory*.3 34) (~is memory*.4 t)) (prn "F - 'new-tagged-value' is the converse of 'maybe-coerce'")) ; Now that we can record types for values we can construct a dynamically typed ; list. (reset) (new-trace "list") ;? (set dump-trace*) (add-fns '((test1 ; 1 points at first node: tagged-value (int 34) ((1 list-address) <- new (list literal)) ((2 tagged-value-address) <- list-value-address (1 list-address)) ((3 type-address) <- get-address (2 tagged-value-address deref) (0 offset)) ((3 type-address deref) <- copy (integer literal)) ((4 location) <- get-address (2 tagged-value-address deref) (1 offset)) ((4 location deref) <- copy (34 literal)) ((5 list-address-address) <- get-address (1 list-address deref) (1 offset)) ((5 list-address-address deref) <- new (list literal)) ; 6 points at second node: tagged-value (boolean t) ((6 list-address) <- copy (5 list-address-address deref)) ((7 tagged-value-address) <- list-value-address (6 list-address)) ((8 type-address) <- get-address (7 tagged-value-address deref) (0 offset)) ((8 type-address deref) <- copy (boolean literal)) ((9 location) <- get-address (7 tagged-value-address deref) (1 offset)) ((9 location deref) <- copy (t literal)) ((10 list-address) <- get (6 list-address deref) (1 offset)) ))) (let first Memory-in-use-until (run 'test1) ;? (prn memory*) (if (or (~all first (map memory* '(1 2 3))) (~is memory*.first 'integer) (~is memory*.4 (+ first 1)) (~is (memory* (+ first 1)) 34) (~is memory*.5 (+ first 2)) (let second memory*.6 (or (~is (memory* (+ first 2)) second) (~all second (map memory* '(6 7 8))) (~is memory*.second 'boolean) (~is memory*.9 (+ second 1)) (~is (memory* (+ second 1)) t) (~is memory*.10 nil)))) (prn "F - lists can contain elements of different types"))) (add-fns '((test2 ((10 list-address) <- list-next (1 list-address))))) (run 'test2) ;? (prn memory*) (if (~is memory*.10 memory*.6) (prn "F - 'list-next can move a list pointer to the next node")) ; 'new-list' takes a variable number of args and constructs a list containing ; them. (reset) (new-trace "new-list") (add-fns '((main ((1 integer) <- new-list (3 literal) (4 literal) (5 literal))))) ;? (set dump-trace*) (run 'main) ;? (prn memory*) (let first memory*.1 ;? (prn first) (if (or (~is memory*.first 'integer) (~is (memory* (+ first 1)) 3) (let second (memory* (+ first 2)) ;? (prn second) (or (~is memory*.second 'integer) (~is (memory* (+ second 1)) 4) (let third (memory* (+ second 2)) ;? (prn third) (or (~is memory*.third 'integer) (~is (memory* (+ third 1)) 5) (~is (memory* (+ third 2) nil))))))) (prn "F - 'new-list' can construct a list of integers"))) ; Just like the table of types is centralized, functions are conceptualized as ; a centralized table of operations just like the 'primitives' we've seen so ; far. If you create a function you can call it like any other op. (reset) (new-trace "new-fn") (add-fns '((test1 ((3 integer) <- add (1 integer) (2 integer))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) (test1)))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4)) (prn "F - calling a user-defined function runs its instructions")) ;? (quit) (reset) (new-trace "new-fn-once") (add-fns '((test1 ((1 integer) <- copy (1 literal))) (main (test1)))) (if (~is 2 (run 'main)) (prn "F - calling a user-defined function runs its instructions exactly once")) ;? (quit) ; User-defined functions communicate with their callers through two ; primitives: ; ; 'arg' - to access inputs ; 'reply' - to return outputs (reset) (new-trace "new-fn-reply") (add-fns '((test1 ((3 integer) <- add (1 integer) (2 integer)) (reply) ((4 integer) <- copy (34 literal))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) (test1)))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4)) (prn "F - 'reply' stops executing the current function")) ;? (quit) (reset) (new-trace "new-fn-reply-nested") (add-fns `((test1 ((3 integer) <- test2)) (test2 (reply (2 integer))) (main ((2 integer) <- copy (34 literal)) (test1)))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 2 34 3 34)) (prn "F - 'reply' stops executing any callers as necessary")) ;? (quit) (reset) (new-trace "new-fn-reply-once") (add-fns '((test1 ((3 integer) <- add (1 integer) (2 integer)) (reply) ((4 integer) <- copy (34 literal))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) (test1)))) (if (~is 4 (run 'main)) ; last reply sometimes not counted. worth fixing? (prn "F - 'reply' executes instructions exactly once")) ;? (quit) (reset) (new-trace "new-fn-arg-sequential") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) <- arg) ((3 integer) <- add (4 integer) (5 integer)) (reply) ((4 integer) <- copy (34 literal))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) (test1 (1 integer) (2 integer)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4 ; add-fn's temporaries 4 1 5 3)) (prn "F - 'arg' accesses in order the operands of the most recent function call (the caller)")) ;? (quit) (reset) (new-trace "new-fn-arg-random-access") (add-fns '((test1 ((5 integer) <- arg 1) ((4 integer) <- arg 0) ((3 integer) <- add (4 integer) (5 integer)) (reply) ((4 integer) <- copy (34 literal))) ; should never run (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) (test1 (1 integer) (2 integer)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4 ; add-fn's temporaries 4 1 5 3)) (prn "F - 'arg' with index can access function call arguments out of order")) ;? (quit) (reset) (new-trace "new-fn-arg-status") (add-fns '((test1 ((4 integer) (5 boolean) <- arg)) (main (test1 (1 literal)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 4 1 5 t)) (prn "F - 'arg' sets a second oarg when arg exists")) ;? (quit) (reset) (new-trace "new-fn-arg-missing") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) <- arg)) (main (test1 (1 literal)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 4 1)) (prn "F - missing 'arg' doesn't cause error")) ;? (quit) (reset) (new-trace "new-fn-arg-missing-2") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) (6 boolean) <- arg)) (main (test1 (1 literal)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 4 1 6 nil)) (prn "F - missing 'arg' wipes second oarg when provided")) ;? (quit) (reset) (new-trace "new-fn-arg-missing-3") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) <- copy (34 literal)) ((5 integer) (6 boolean) <- arg)) (main (test1 (1 literal)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 4 1 6 nil)) (prn "F - missing 'arg' consistently wipes its oarg")) ;? (quit) (reset) (new-trace "new-fn-arg-missing-3") (add-fns '((test1 ; if given two args, adds them; if given one arg, increments ((4 integer) <- arg) ((5 integer) (6 boolean) <- arg) { begin (break-if (6 boolean)) ((5 integer) <- copy (1 literal)) } ((7 integer) <- add (4 integer) (5 integer))) (main (test1 (34 literal)) ))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 4 34 5 1 6 nil 7 35)) (prn "F - function with optional second arg")) ;? (quit) ; how should errors be handled? will be unclear until we support concurrency and routine trees. (reset) (new-trace "new-fn-reply-oarg") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) <- arg) ((6 integer) <- add (4 integer) (5 integer)) (reply (6 integer)) ((4 integer) <- copy (34 literal))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) ((3 integer) <- test1 (1 integer) (2 integer))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4 ; add-fn's temporaries 4 1 5 3 6 4)) (prn "F - 'reply' can take aguments that are returned, or written back into output args of caller")) (reset) (new-trace "new-fn-reply-oarg-multiple") (add-fns '((test1 ((4 integer) <- arg) ((5 integer) <- arg) ((6 integer) <- add (4 integer) (5 integer)) (reply (6 integer) (5 integer)) ((4 integer) <- copy (34 literal))) (main ((1 integer) <- copy (1 literal)) ((2 integer) <- copy (3 literal)) ((3 integer) (7 integer) <- test1 (1 integer) (2 integer))))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 1 2 3 3 4 7 3 ; add-fn's temporaries 4 1 5 3 6 4)) (prn "F - 'reply' permits a function to return multiple values at once")) ; Our control operators are quite inconvenient to use, so mu provides a ; lightweight tool called 'convert-braces' to work in a slightly more ; convenient format with nested braces: ; ; { ; some instructions ; { ; more instructions ; } ; } ; ; Braces are just labels, they require no special parsing. The operations ; 'break' and 'continue' jump to just after the enclosing '}' and '{' ; respectively. ; ; Conditional and unconditional 'break' and 'continue' should give us 80% of ; the benefits of the control-flow primitives we're used to in other ; languages, like 'if', 'while', 'for', etc. (reset) (new-trace "convert-braces") (if (~iso (convert-braces '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) { begin ; 'begin' is just a hack because racket turns curlies into parens ((4 boolean) <- neq (1 integer) (3 integer)) (break-if (4 boolean)) ((5 integer) <- copy (34 literal)) } (reply))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) ((4 boolean) <- neq (1 integer) (3 integer)) (jump-if (4 boolean) (1 offset)) ((5 integer) <- copy (34 literal)) (reply))) (prn "F - convert-braces replaces break-if with a jump-if to after the next close curly")) (reset) (new-trace "convert-braces-empty-block") (if (~iso (convert-braces '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) { begin (break) } (reply))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) (jump (0 offset)) (reply))) (prn "F - convert-braces works for degenerate blocks")) (reset) (new-trace "convert-braces-nested-break") (if (~iso (convert-braces '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) { begin ((4 boolean) <- neq (1 integer) (3 integer)) (break-if (4 boolean)) { begin ((5 integer) <- copy (34 literal)) } } (reply))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) ((4 boolean) <- neq (1 integer) (3 integer)) (jump-if (4 boolean) (1 offset)) ((5 integer) <- copy (34 literal)) (reply))) (prn "F - convert-braces balances curlies when converting break")) (reset) (new-trace "convert-braces-nested-continue") (if (~iso (convert-braces '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) { begin ((3 integer) <- add (2 integer) (2 integer)) { begin ((4 boolean) <- neq (1 integer) (3 integer)) } (continue-if (4 boolean)) ((5 integer) <- copy (34 literal)) } (reply))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (2 integer) (2 integer)) ((4 boolean) <- neq (1 integer) (3 integer)) (jump-if (4 boolean) (-3 offset)) ((5 integer) <- copy (34 literal)) (reply))) (prn "F - convert-braces balances curlies when converting continue")) (reset) (new-trace "continue") ;? (set dump-trace*) (add-fns '((main ((1 integer) <- copy (4 literal)) ((2 integer) <- copy (1 literal)) { begin ((2 integer) <- add (2 integer) (2 integer)) ((3 boolean) <- neq (1 integer) (2 integer)) (continue-if (3 boolean)) ((4 integer) <- copy (34 literal)) } (reply)))) ;? (each stmt function*!main ;? (prn stmt)) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 4 2 4 3 nil 4 34)) (prn "F - continue correctly loops")) ; todo: fuzz-test invariant: convert-braces offsets should be robust to any ; number of inner blocks inside but not around the continue block. (reset) (new-trace "continue-nested") ;? (set dump-trace*) (add-fns '((main ((1 integer) <- copy (4 literal)) ((2 integer) <- copy (1 literal)) { begin ((2 integer) <- add (2 integer) (2 integer)) { begin ((3 boolean) <- neq (1 integer) (2 integer)) } (continue-if (3 boolean)) ((4 integer) <- copy (34 literal)) } (reply)))) ;? (each stmt function*!main ;? (prn stmt)) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 4 2 4 3 nil 4 34)) (prn "F - continue correctly loops")) (reset) (new-trace "continue-fail") (add-fns '((main ((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) { begin ((2 integer) <- add (2 integer) (2 integer)) { begin ((3 boolean) <- neq (1 integer) (2 integer)) } (continue-if (3 boolean)) ((4 integer) <- copy (34 literal)) } (reply)))) (run 'main) ;? (prn memory*) (if (~iso memory* (obj 1 4 2 4 3 nil 4 34)) (prn "F - continue might never trigger")) ; using tagged-values you can define generic functions that run different code ; based on the types of their args. (reset) (new-trace "dispatch-clause") ;? (set dump-trace*) (add-fns '((test1 ((4 tagged-value-address) <- arg) { begin ((5 integer) (6 boolean) <- maybe-coerce (4 tagged-value-address deref) (integer literal)) (break-unless (6 boolean)) ((7 tagged-value-address) <- arg) ((8 integer) (9 boolean) <- maybe-coerce (7 tagged-value-address deref) (integer literal)) ((9 integer) <- add (5 integer) (8 integer)) (reply (9 integer)) } (reply (nil literal))) (main ((1 tagged-value-address) <- new-tagged-value (integer literal) (34 literal)) ((2 tagged-value-address) <- new-tagged-value (integer literal) (3 literal)) ((3 integer) <- test1 (1 tagged-value-address) (2 tagged-value-address))))) (run 'main) ;? (prn memory*) (if (~is memory*.3 37) (prn "F - an example function that checks that its oarg is an integer")) ;? (quit) ; todo - test that reply increments pc for caller frame after popping current frame (reset) (new-trace "dispatch-multiple-clauses") ;? (set dump-trace*) (add-fns '((test1 ((4 tagged-value-address) <- arg) { begin ((5 integer) (6 boolean) <- maybe-coerce (4 tagged-value-address deref) (integer literal)) (break-unless (6 boolean)) ((7 tagged-value-address) <- arg) ((8 integer) (9 boolean) <- maybe-coerce (7 tagged-value-address deref) (integer literal)) ((9 integer) <- add (5 integer) (8 integer)) (reply (9 integer)) } { begin ((5 boolean) (6 boolean) <- maybe-coerce (4 tagged-value-address deref) (boolean literal)) (break-unless (6 boolean)) ((7 tagged-value-address) <- arg) ((8 boolean) (9 boolean) <- maybe-coerce (7 tagged-value-address deref) (boolean literal)) ((9 boolean) <- or (5 boolean) (8 boolean)) (reply (9 boolean)) } (reply (nil literal))) (main ((1 tagged-value-address) <- new-tagged-value (boolean literal) (t literal)) ((2 tagged-value-address) <- new-tagged-value (boolean literal) (nil literal)) ((3 boolean) <- test1 (1 tagged-value-address) (2 tagged-value-address))))) ;? (each stmt function*!test-fn ;? (prn " " stmt)) (run 'main) ;? (wipe dump-trace*) ;? (prn memory*) (if (~is memory*.3 t) (prn "F - an example function that can do different things (dispatch) based on the type of its args or oargs")) ;? (quit) (reset) (new-trace "dispatch-multiple-calls") (add-fns '((test1 ((4 tagged-value-address) <- arg) { begin ((5 integer) (6 boolean) <- maybe-coerce (4 tagged-value-address deref) (integer literal)) (break-unless (6 boolean)) ((7 tagged-value-address) <- arg) ((8 integer) (9 boolean) <- maybe-coerce (7 tagged-value-address deref) (integer literal)) ((9 integer) <- add (5 integer) (8 integer)) (reply (9 integer)) } { begin ((5 boolean) (6 boolean) <- maybe-coerce (4 tagged-value-address deref) (boolean literal)) (break-unless (6 boolean)) ((7 tagged-value-address) <- arg) ((8 boolean) (9 boolean) <- maybe-coerce (7 tagged-value-address deref) (boolean literal)) ((9 boolean) <- or (5 boolean) (8 boolean)) (reply (9 boolean)) } (reply (nil literal))) (main ((1 tagged-value-address) <- new-tagged-value (boolean literal) (t literal)) ((2 tagged-value-address) <- new-tagged-value (boolean literal) (nil literal)) ((3 boolean) <- test1 (1 tagged-value-address) (2 tagged-value-address)) ((10 tagged-value-address) <- new-tagged-value (integer literal) (34 literal)) ((11 tagged-value-address) <- new-tagged-value (integer literal) (3 literal)) ((12 integer) <- test1 (10 tagged-value-address) (11 tagged-value-address))))) (run 'main) ;? (prn memory*) (if (~and (is memory*.3 t) (is memory*.12 37)) (prn "F - different calls can exercise different clauses of the same function")) ; A rudimentary memory allocator. Eventually we want to write this in mu. (reset) (new-trace "new-primitive") (add-fns '((main ((1 integer-address) <- new (integer literal))))) (let before Memory-in-use-until (run 'main) ;? (prn memory*) (if (~iso memory*.1 before) (prn "F - 'new' returns current high-water mark")) (if (~iso Memory-in-use-until (+ before 1)) (prn "F - 'new' on primitive types increments high-water mark by their size"))) (reset) (new-trace "new-array-literal") (add-fns '((main ((1 type-array-address) <- new (type-array literal) (5 literal))))) (let before Memory-in-use-until (run 'main) ;? (prn memory*) (if (~iso memory*.1 before) (prn "F - 'new' on array with literal size returns current high-water mark")) (if (~iso Memory-in-use-until (+ before 6)) (prn "F - 'new' on primitive arrays increments high-water mark by their size"))) (reset) (new-trace "new-array-direct") (add-fns '((main ((1 integer) <- copy (5 literal)) ((2 type-array-address) <- new (type-array literal) (1 integer))))) (let before Memory-in-use-until (run 'main) ;? (prn memory*) (if (~iso memory*.2 before) (prn "F - 'new' on array with variable size returns current high-water mark")) (if (~iso Memory-in-use-until (+ before 6)) (prn "F - 'new' on primitive arrays increments high-water mark by their (variable) size"))) ; A rudimentary process scheduler. You can 'run' multiple functions at once, ; and they share the virtual processor. ; There's also a 'fork' primitive to let functions create new threads of ; execution (we call them routines). ; Eventually we want to allow callers to influence how much of their CPU they ; give to their 'children', or to rescind a child's running privileges. (reset) (new-trace "scheduler") (add-fns '((f1 ((1 integer) <- copy (3 literal))) (f2 ((2 integer) <- copy (4 literal))))) (let ninsts (run 'f1 'f2) (when (~iso 2 ninsts) (prn "F - scheduler didn't run the right number of instructions: " ninsts))) (if (~iso memory* (obj 1 3 2 4)) (prn "F - scheduler runs multiple functions: " memory*)) (check-trace-contents "scheduler orders functions correctly" '(("schedule" "f1") ("schedule" "f2") )) (check-trace-contents "scheduler orders schedule and run events correctly" '(("schedule" "f1") ("run" "f1 0") ("schedule" "f2") ("run" "f2 0") )) ; The scheduler needs to keep track of the call stack for each routine. ; Eventually we'll want to save this information in mu's address space itself, ; along with the types array, the magic buffers for args and oargs, and so on. ; ; Eventually we want the right stack-management primitives to build delimited ; continuations in mu. ; Routines can throw errors. (reset) (new-trace "array-bounds-check") (add-fns '((main ((1 integer) <- copy (2 literal)) ((2 integer) <- copy (23 literal)) ((3 integer) <- copy (24 literal)) ((4 integer) <- index (1 integer-array) (2 literal))))) ;? (set dump-trace*) (run 'main) ;? (prn memory*) (let last-routine (deq completed-routines*) (if (no rep.last-routine!error) (prn "F - 'index' throws an error if out of bounds"))) ; --- (reset) (new-trace "convert-names-local") (if (~iso (convert-names '(((x integer) <- copy (4 literal)) ((y integer) <- copy (2 literal)) ((z integer) <- add (x integer) (y integer)))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((3 integer) <- add (1 integer) (2 integer)))) (prn "F - convert-names renames symbolic names to integer offsets")) (reset) (new-trace "convert-names-nil") (if (~iso (convert-names '(((x integer) <- copy (4 literal)) ((y integer) <- copy (2 literal)) ((nil integer) <- add (x integer) (y integer)))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((nil integer) <- add (1 integer) (2 integer)))) (prn "F - convert-names never renames nil")) (reset) (new-trace "convert-quotes-defer") (if (~iso (convert-quotes '(((1 integer) <- copy (4 literal)) (defer [ ((3 integer) <- copy (6 literal)) ]) ((2 integer) <- copy (5 literal)))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (5 literal)) ((3 integer) <- copy (6 literal)))) (prn "F - convert-quotes can handle 'defer'")) (reset) (new-trace "set-default-scope") (add-fns '((main ((default-scope scope-address) <- new (scope literal) (2 literal)) ((1 integer) <- copy (23 literal))))) (let before Memory-in-use-until ;? (set dump-trace*) (run 'main) ;? (prn memory*) (if (~and (~is 23 memory*.1) (is 23 (memory* (+ before 1)))) (prn "F - default-scope implicitly modifies variable locations"))) (reset) (new-trace "set-default-scope-skips-offset") (add-fns '((main ((default-scope scope-address) <- new (scope literal) (2 literal)) ((1 integer) <- copy (23 offset))))) (let before Memory-in-use-until ;? (set dump-trace*) (run 'main) ;? (prn memory*) (if (~and (~is 23 memory*.1) (is 23 (memory* (+ before 1)))) (prn "F - default-scope skips 'offset' types just like literals"))) (reset) (new-trace "default-scope-bounds-check") (add-fns '((main ((default-scope scope-address) <- new (scope literal) (2 literal)) ((2 integer) <- copy (23 literal))))) ;? (set dump-trace*) (run 'main) ;? (prn memory*) (let last-routine (deq completed-routines*) (if (no rep.last-routine!error) (prn "F - default-scope checks bounds"))) (reset) (new-trace "convert-names-default-scope") (if (~iso (convert-names '(((x integer) <- copy (4 literal)) ((y integer) <- copy (2 literal)) ((default-scope integer) <- add (x integer) (y integer)))) '(((1 integer) <- copy (4 literal)) ((2 integer) <- copy (2 literal)) ((default-scope integer) <- add (1 integer) (2 integer)))) (prn "F - convert-names never renames default-scope")) (reset) (new-trace "suppress-default-scope") (add-fns '((main ((default-scope scope-address) <- new (scope literal) (2 literal)) ((1 integer global) <- copy (23 literal))))) (let before Memory-in-use-until ;? (set dump-trace*) (run 'main) ;? (prn memory*) (if (~and (is 23 memory*.1) (~is 23 (memory* (+ before 1)))) (prn "F - default-scope skipped for locations with metadata 'global'"))) (reset) (new-trace "convert-names-global") (if (~iso (convert-names '(((x integer) <- copy (4 literal)) ((y integer global) <- copy (2 literal)) ((default-scope integer) <- add (x integer) (y integer global)))) '(((1 integer) <- copy (4 literal)) ((y integer global) <- copy (2 literal)) ((default-scope integer) <- add (1 integer) (y integer global)))) (prn "F - convert-names never renames global operands")) (reset) ; end file with this to persist the trace for the final test