| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
Turns out the slowdown reported in 3309 was almost entirely due to
commit 3305: supporting extremely small floating point numbers.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rip out everything to fix one failing unit test (commit 3290; type
abbreviations).
This commit does several things at once that I couldn't come up with a
clean way to unpack:
A. It moves to a new representation for type trees without changing
the actual definition of the `type_tree` struct.
B. It adds unit tests for our type metadata precomputation, so that
errors there show up early and in a simpler setting rather than dying
when we try to load Mu code.
C. It fixes a bug, guarding against infinite loops when precomputing
metadata for recursive shape-shifting containers. To do this it uses a
dumb way of comparing type_trees, comparing their string
representations instead. That is likely incredibly inefficient.
Perhaps due to C, this commit has made Mu incredibly slow. Running all
tests for the core and the edit/ app now takes 6.5 minutes rather than
3.5 minutes.
== more notes and details
I've been struggling for the past week now to back out of a bad design
decision, a premature optimization from the early days: storing atoms
directly in the 'value' slot of a cons cell rather than creating a
special 'atom' cons cell and storing it on the 'left' slot. In other
words, if a cons cell looks like this:
o
/ | \
left val right
..then the type_tree (a b c) used to look like this (before this
commit):
o
| \
a o
| \
b o
| \
c null
..rather than like this 'classic' approach to s-expressions which never
mixes val and right (which is what we now have):
o
/ \
o o
| / \
a o o
| / \
b o null
|
c
The old approach made several operations more complicated, most recently
the act of replacing a (possibly atom/leaf) sub-tree with another. That
was the final straw that got me to realize the contortions I was going
through to save a few type_tree nodes (cons cells).
Switching to the new approach was hard partly because I've been using
the old approach for so long and type_tree manipulations had pervaded
everything. Another issue I ran into was the realization that my layers
were not cleanly separated. Key parts of early layers (precomputing type
metadata) existed purely for far later ones (shape-shifting types).
Layers I got repeatedly stuck at:
1. the transform for precomputing type sizes (layer 30)
2. type-checks on merge instructions (layer 31)
3. the transform for precomputing address offsets in types (layer 36)
4. replace operations in supporting shape-shifting recipes (layer 55)
After much thrashing I finally noticed that it wasn't the entirety of
these layers that was giving me trouble, but just the type metadata
precomputation, which had bugs that weren't manifesting until 30 layers
later. Or, worse, when loading .mu files before any tests had had a
chance to run. A common failure mode was running into types at run time
that I hadn't precomputed metadata for at transform time.
Digging into these bugs got me to realize that what I had before wasn't
really very good, but a half-assed heuristic approach that did a whole
lot of extra work precomputing metadata for utterly meaningless types
like `((address number) 3)` which just happened to be part of a larger
type like `(array (address number) 3)`.
So, I redid it all. I switched the representation of types (because the
old representation made unit tests difficult to retrofit) and added unit
tests to the metadata precomputation. I also made layer 30 only do the
minimal metadata precomputation it needs for the concepts introduced
until then. In the process, I also made the precomputation more correct
than before, and added hooks in the right place so that I could augment
the logic when I introduced shape-shifting containers.
== lessons learned
There's several levels of hygiene when it comes to layers:
1. Every layer introduces precisely what it needs and in the simplest
way possible. If I was building an app until just that layer, nothing
would seem over-engineered.
2. Some layers are fore-shadowing features in future layers. Sometimes
this is ok. For example, layer 10 foreshadows containers and arrays and
so on without actually supporting them. That is a net win because it
lets me lay out the core of Mu's data structures out in one place. But
if the fore-shadowing gets too complex things get nasty. Not least
because it can be hard to write unit tests for features before you
provide the plumbing to visualize and manipulate them.
3. A layer is introducing features that are tested only in later layers.
4. A layer is introducing features with tests that are invalidated in
later layers. (This I knew from early on to be an obviously horrendous
idea.)
Summary: avoid Level 2 (foreshadowing layers) as much as possible.
Tolerate it indefinitely for small things where the code stays simple
over time, but become strict again when things start to get more
complex.
Level 3 is mostly a net lose, but sometimes it can be expedient (a real
case of the usually grossly over-applied term "technical debt"), and
it's better than the conventional baseline of no layers and no
scenarios. Just clean it up as soon as possible.
Definitely avoid layer 4 at any time.
== minor lessons
Avoid unit tests for trivial things, write scenarios in context as much as
possible. But within those margins unit tests are fine. Just introduce them
before any scenarios (commit 3297).
Reorganizing layers can be easy. Just merge layers for starters! Punt on
resplitting them in some new way until you've gotten them to work. This is the
wisdom of Refactoring: small steps.
What made it hard was not wanting to merge *everything* between layer 30
and 55. The eventual insight was realizing I just need to move those two
full-strength transforms and nothing else.
|
| | |
|
| | |
|
| |
|
|
| |
Thanks Ella Couch for finding this.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Thanks Ella Couch for pointing out that Mu was lying when debugging
small numbers.
def main [
local-scope
x:number <- copy 1
{
x <- divide x, 2
$print x, 10/newline
loop # until SIGFPE
}
]
|
| |
|
|
|
|
| |
Highlight a couple of places where it turns out that we're flying by the
seat of our pants with heuristics, and we don't really understand how to
precompute metadata for a program's types.
|
| |
|
|
| |
Correction for syntax highlighting.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
I've been putting up for some time with the tension between wanting to
show scenarios at the top of the layer even if I want to *run* any unit
tests of sub-components introduced within the layer before them. Turned
out to be an easy fix.
We don't have very many of these, and the unit tests in the early layers
don't compete with any scenarios, so I don't need to mess with them. But
this is a key tool in my toolkit, to be able to decouple presentation
order from run order for tests.
Though now the separate compilation units are again unbalanced; sigh.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Stop double-counting failing tests in some situations.
|
| |
|
|
| |
New test not fully passing yet.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For the last couple of days I've been implicitly thinking in terms of
how many compilation units I want to generate. Might as well make that
explicit and drop the hacky ideas for approximating it.
I tried more timing experiments like the ones in commit 3281.
Conclusion: I can't have the best of both worlds:
1. Full compilation doesn't take too much longer than with a single
compilation unit.
2. Incremental compilation is fast enough that there's negligible
benefit from dropping optimization.
We're still taking on a 10s hit in full build time.
I care more about not degrading the full compilation too much, since
that gets magnified so much on the Couch's puny server. So we'll just
have to continue using CXXFLAGS=-g when we care to save a few seconds in
incremental compilation time.
A final mystery: the build time increases by 10s with the new heuristic
even though the number of calls to the compiler (and therefore the fixed
cost) is the same. Seems like separating certain functions into
different units is causing the compiler issues. Dropping from 4 to 3
compilation units eliminated the issue.
--- Appendix: Measurements
before:
full build 4 + test: 42s
incremental compilation with -O3: varied from 30s for mu_0.cc to 5s for mu_3.cc
longer times benefitted from dropping -O3
after:
full build 1 + test: 39s
full build 2 + test: 41s
full build 3 + test: 43s
full build 4 + test: 52s
full build 5 + test: 53s
full build 6 + test: 51s
full build 10 (9) + test: 54s
full build 20 (16) + test: 58s
|
| |
|
|
|
|
|
|
|
|
| |
Now that we have a new build system we shouldn't need to run unoptimized
just to save time. (Though that's not strictly true; if a change
modifies .build/mu_0.cc which is twice as large as later compilation
units, dropping -O3 shaves 10s off the time for an incremental build.)
Since we don't need to run unoptimized anymore, let's just explicitly
ask for --test-only-app when we need it.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Fix CI.
|
| |
|
|
|
|
|
|
|
| |
Fix CI process after recent changes. CI still will not be actually
*making use* of separate compilation (as it shouldn't).
As a side effect, 'build_until' shows a simpler (but still working!)
process for building Mu. Vast improvement over the previous hack of
dipping selectively into the Makefile.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before:
layers -> tangle -> g++
All changes to (C++) layers triggered recompilation of everything,
taking 35s on my laptop, and over 4 minutes on a puny server with just
512MB of RAM.
After:
layers -> tangle -> cleave -> g++
Now a tiny edit takes just 5s to recompile on my laptop.
My initial approach was to turn each function into a separate
compilation unit under the .build/ directory. That blew up the time for
a full/initial compilation to almost 6 minutes on my laptop. Trial and
error showed 4 compilation units to be close to the sweet spot. Full
compilation is still slightly slower (43s) but not by much.
I could speed things up further by building multiple of the compilation
units in parallel (the recursive invocation in 'makefile'). But that
would put more pressure on a puny server, so I'm going to avoid getting
too aggressive.
--- Other considerations
I spent some time manually testing the dependency structure to the
makefile, making sure that files aren't unnecessarily written to disk,
modifying their timestamp and triggering dependent work; that changes to
layers don't unnecessarily modify the common headers or list of globals;
that changes to the cleave/ tool itself rebuild the entire project; that
the old auto-generated '_list' files plug in at the right stage in the
pipeline; that changes to common headers trigger recompilation of
everything; etc. Too bad it's not easy to write some tests for all this.
I spent some time trying to make sure the makefile was not too opaque to
a newcomer. The targets mostly flow from top to bottom. There's a little
diagram at the top that is hopefully illuminating. When I had 700
compilation units for 700 functions I stopped printing each one of those
compilation commands, but when I backed off to just 4 compilation units
I decided to err on the side of making the build steps easy to see.
|
| | |
|
| |
|
|
|
| |
Stop inlining functions because that will complicate separate
compilation. It also simplifies the code without impacting performance.
|
| | |
|
| |
|
|
| |
Streamline the build process a bit.
|
| | |
|
| |
|
|
| |
Follow convention more closely by using CXXFLAGS for C++ files.
|
| |
|
|
| |
Always keep macro definitions in the Includes section.
|
| |
|
|
|
|
|
|
|
|
|
| |
Undo 3272. The trouble with creating a new section for constants is that
there's no good place to order it since constants can be initialized
using globals as well as vice versa. And I don't want to add constraints
disallowing either side.
Instead, a new plan: always declare constants in the Globals section
using 'extern const' rather than just 'const', since otherwise constants
implicitly have internal linkage (http://stackoverflow.com/questions/14894698/why-does-extern-const-int-n-not-work-as-expected)
|
| |
|
|
|
|
| |
Move global constants into their own section since we seem to be having
trouble linking in 'extern const' variables when manually cleaving mu.cc
into separate compilation units.
|
| |
|
|
| |
Disallow defining multiple globals at once.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Clean up the Globals section so that we can generate extern declarations
for all globals out using this command after we carve it out into
globals.cc:
grep ';' globals.cc |perl -pwe 's/[=(].*/;/' |perl -pwe 's/^[^\/# ]/extern $&/' > globals.h
The first perl command strips out initializers. The second prepends
'extern'. This simplistic approach requires each global definition to
lie all on one line.
|
| |
|
|
|
|
|
|
|
| |
Deconstruct the tracing layer which had been an exception to our
includes-types-prototypes-globals-functions organization thus far.
To do this we predefine a few primitive globals before the types that
use them, and we pull some method definitions out of struct definitions
at the cost of having to manually write a couple of prototypes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Right now Mu has zero dependency knowledge. If anything changes in our
project the C++ compiler has to redo the entire project. This is
unnecessarily slow, and also causes gcc to run out of RAM on puny
machines.
New vision: carve the tangled mu.cc into multiple files.
includes.h
types.h
globals.cc
globals.h
one .cc file for each function definition
(This is of course in addition to the already auto-generated test_list
and function_list.)
With this approach changes to functions will only require recompiling
the functions that changed. We'd need to be smart to not rewrite files
that don't change (modulo #line directives).
Any changes to includes/types/globals would still require rebuilding the
entire project. That's the (now greatly reduced) price we will continue
to pay for outsourcing dependency management to the computer.
Plan arrived at after conversation with Stephen Malina.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Ah, the reason commit 3258 broke chessboard.mu was that I forgot to
migrate the implementation of 'switch' to 'wait-for-routine-to-block'.
That caused these cascading effects when running chessboard.mu:
a) 'read-event' from real keyboard calls 'switch'
b) 'switch' waits for some other currently running routine to *complete*
rather than just block
c) deadlock unsurprisingly ensues
This was hard to debug because I kept searching for occurrences of
'wait-for-routine' that I'd missed, and didn't realize that 'switch' too
was a form of 'wait-for-routine'. No more; now it's a form of
'wait-for-routine-to-block', possibly the *only* reason to ever call
that instruction outside of tests.
|
| |
|
|
|
|
| |
Turns out chessboard.mu started deadlocking in commit 3258 even though
all its tests continue to pass. Not fixed yet; first make deadlock
easier to diagnose.
|
| |
|
|
| |
Commit 3171 which added '--trace' broke 'Save_trace'.
|
| | |
|
| |
|
|
| |
Fix CI.
|