aboutsummaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorbptato <nincsnevem662@gmail.com>2024-11-14 19:41:33 +0100
committerbptato <nincsnevem662@gmail.com>2024-11-14 19:47:56 +0100
commitfd10019d8b8a1b5dcb0026521fa8f5f072c3a10a (patch)
treeb57b36570bb4d5e119485718e70bdcb71a68d9e0 /doc
parent352b21b7fed0c0ca5e19952f1f0e1dab03b2c140 (diff)
downloadchawan-fd10019d8b8a1b5dcb0026521fa8f5f072c3a10a.tar.gz
Update docs
Diffstat (limited to 'doc')
-rw-r--r--doc/architecture.md391
1 files changed, 202 insertions, 189 deletions
diff --git a/doc/architecture.md b/doc/architecture.md
index 81f18413..e67b9170 100644
--- a/doc/architecture.md
+++ b/doc/architecture.md
@@ -16,36 +16,38 @@ This document describes some aspects of how Chawan works.
* [General](#general)
* [JS in the pager](#js-in-the-pager)
* [JS in the buffer](#js-in-the-buffer)
-* [Styling](#styling)
-* [Layout](#layout)
-* [Rendering](#rendering)
+* [CSS](#css)
+ * [Parsing, cascading](#parsing-cascading)
+ * [Layout](#layout)
+ * [Rendering](#rendering)
## Module organization
Explanation for the separate directories found in `src/`:
* config: configuration-related code. Mainly parsers for config files.
-* css: styling-related code; CSS parsing and cascading.
-* html: DOM building, DOM functions, the DOM itself, forms, etc. Note that it
- does not include the [HTML parser](https://git.sr.ht/~bptato/chame) itself.
+* css: CSS parsing, cascading, layout, rendering.
+* html: DOM building, DOM functions, the DOM itself, forms, etc. It does
+ not include the [HTML parser](https://git.sr.ht/~bptato/chame) itself.
* io: code for IPC, interaction with the file system, etc.
* js: modules mainly for use by JS code.
-* layout: the layout engine and its renderer.
* loader: code for the file loader server (?).
* local: code for the main process (i.e. the pager).
-* server: code for processes other than the main process, e.g. buffer and
- forkserver. (Why is loader not here? idk)
-* types: mainly definitions of data types and things I didn't know where to put.
+* server: code for processes other than the main process, e.g. buffer
+ and forkserver. (Why is loader not here? idk)
+* types: mainly definitions of data types and things I didn't know where
+ to put.
* utils: things I didn't know where to put part 2
Additionally, "adapters" of various protocols and file formats can be found in
`adapter/`:
* protocol: includes support for every protocol supported by Chawan.
-* format: HTML converters for various text-based file formats.
-* img: image decoders and encoders. In general, these just read and output
- RGBA data through standard I/O (which may actually be a cache file; see the
- [image docs](image.md) for details).
+* format: HTML converters for various text-based file formats,
+ e.g. Markdown.
+* img: image decoders and encoders. In general, these just read and
+ output RGBA data through standard I/O (which may actually be a cache
+ file; see the [image docs](image.md) for details).
## Process model
@@ -91,171 +93,178 @@ space is replaced by exec anyway. (Also, it would be slow.)
### Loader
The loader process takes requests from the main process and the buffer
-processes. Then, depending on the scheme, it responds by performing one of the
+processes. Then, depending on the scheme, it performs one of the
following steps:
-* `cgi-bin:` Start a CGI script, and read out its stdout into the response body.
- For the main contents of text-based buffers, it also saves the response
- into the cache.
- This is also used for schemes like http/s, ftp, etc. by internally rewriting
- them into the appropriate `cgi-bin:` URL.
-* `stream:` Do the same thing as above, but read from a file descriptor passed
- to the loader beforehand. This is used when stdin is a file, e.g.
- `echo test | cha`. It is also used for mailcap entries with an x-htmloutput
- field.
-* `cache:` Read the file from the cache. This is used by the pager for the
- "view source" operation, and by buffers in the rare situation where their
- initial character encoding guess proves to be incorrect and they need to
- rewind the source.
+* `cgi-bin:` Start a CGI script, and read out its stdout into the
+ response body. In certain cases it also streams the response into
+ the cache.
+ This is also used for schemes like http/s, ftp, etc. by internally
+ rewriting them into the appropriate `cgi-bin:` URL.
+* `stream:` Do the same thing as above, but read from a file descriptor
+ passed to the loader beforehand. This is used when stdin is a file,
+ e.g. `echo test | cha`. It is also used for mailcap entries with an
+ x-htmloutput field.
+* `cache:` Read the file from the cache. This is used by the pager
+ for the "view source" operation, and by buffers in the rare situation
+ where their initial character encoding guess proves to be incorrect
+ and they need to rewind the source.
* `data:` Decode a data URL. This is done directly in the loader process
because very long data URLs wouldn't fit into the environment. (Plus,
obviously, it's more efficient this way.)
-The loader process distinguishes between clients (i.e the main process or
-buffers) through client keys. In theory this should help against rogue clients,
-though in practice it is still trivial to crash the loader as a client. It also
-helps us block further requests from buffers that have been discarded by the
-pager, but still haven't found out yet that their life time has ended.
+The loader process distinguishes between clients (i.e processes) through
+client keys (session cookies). In theory this should help against rogue
+clients, though in practice it is still trivial to crash the loader as
+a client. It also helps us block further requests from buffers that have
+been discarded by the pager, but still haven't found out yet that their
+life time has ended.
### Buffer
-Buffer processes parse HTML, optionally query external resources from loader,
-run styling, JS, and finally render the page to an internal canvas.
+Buffer processes parse HTML, optionally query external resources from
+loader, run styling, JS, and finally render the page to an internal
+canvas.
-Buffers are managed by the pager through Container objects. A UNIX domain socket
-is established between each buffer and the pager to enable communication between
-them.
+Buffers are managed by the pager through Container objects. A UNIX
+domain socket is established between each buffer and the pager for
+IPC.
## Opening buffers
Scenario: the user attempts to navigate to <https://example.org>.
1. pager creates a new container for the target URL.
-2. pager sends a request for "https://example.org" to the loader. Then, it
- registers the file descriptor in its selector, and does something else until
- poll() reports activity on the file descriptor.
-3. loader rewrites "https://example.org" into "cgi-bin:http". It then runs the
- http CGI script with the appropriate environment variables set to parts of
- this URL and request headers.
-4. The http CGI script opens a connection to example.org. When connected, it
- starts printing out headers it receives to stdout.
+2. pager sends a request for "https://example.org" to the loader. Then,
+ it registers the file descriptor in its selector, and does something
+ else until poll() reports activity on the file descriptor.
+3. loader rewrites "https://example.org" into "cgi-bin:http". It then
+ runs the http CGI script with the appropriate environment variables
+ set to parts of this URL and request headers.
+4. The http CGI script opens a connection to example.org. When
+ connected, it starts writing headers it receives to stdout.
5. loader parses these headers, and sends them to pager.
-6. pager reads in the headers, and decides what to do based on the Content-Type.
- * If Content-Type is found in mailcap, then the command in that
- mailcap entry is executed, with the response body dup2'd onto its
- stdin. If the entry has x-htmloutput, then the command's stdout is
- taken instead of the response body, and Content-Type is set to
+6. pager reads in the headers, and decides what to do based on the
+ Content-Type:
+ * If Content-Type is found in mailcap, then the response body
+ is piped into the command in that mailcap entry. If the
+ entry has x-htmloutput, then the command's stdout is taken
+ instead of the response body, and Content-Type is set to
text/html. Otherwise, the container is discarded.
- * If Content-Type is text/html, then a new buffer process is created,
- which then parses the response body as HTML. If it is any `text/*`
- subtype, then the response is simply inserted into a `<plaintext>` tag.
- * If Content-Type is not a `text/*` subtype, and no mailcap entry for it
- is found, then the user is prompted about where they wish to save the
- file.
+ * If Content-Type is text/html, then a new buffer process is
+ created, which then parses the response body as HTML. If it
+ is any `text/*` subtype, then the response is simply inserted
+ into a `<plaintext>` tag.
+ * If Content-Type is not a `text/*` subtype, and no mailcap
+ entry for it is found, then the user is prompted about where
+ they wish to save the file.
## Cache
-Chawan's caching mechanism is largely inspired by that of w3m, which does not
-have a network cache. Instead, it simply saves source files to the disk before
-displaying them, and lets users view/edit the source without another network
-request.
+Chawan's caching mechanism is largely inspired by that of w3m, which
+does not have a network cache. Instead, it simply saves source files
+to the disk before displaying them, and lets users view/edit the source
+without another network request.
-Chawan improves upon this by simultaneously streaming files to the cache and
-buffers:
+The only difference in Chawan is that it simultaneously streams files
+to the cache *and* buffers:
-1. Client (pager or buffer) initiates request by sending a message to loader.
+1. Client (pager or buffer) initiates request by sending a message to
+ loader.
2. Loader starts CGI script, reads headers, sends a response, and waits.
-3. Client now may send an "addCacheFile" message, which prompts loader to add a
- cache file for this request.
-4. Client sends resume, now loader will stream the response both to the client
- and the cache.
+3. Client now may send an "addCacheFile" message, which prompts loader
+ to add a cache file for this request.
+4. Client sends "resume", now loader will stream the response both to
+ the client and the cache.
-Cached items may be shared between clients; this is how rewinding on wrong
-charset guess is implemented. They are also manually reference counted and are
-unlinked when their reference count drops to zero.
+Cached items may be shared between clients; this is how rewinding on
+wrong charset guess is implemented. They are also manually reference
+counted and are unlinked when their reference count drops to zero.
The cache is used in the following ways:
* For view source and edit source operations.
-* For rewinding buffers on incorrect charset guess. (In practice, this is almost
- never used, because the first chunk we read tends to determine the charset
- unambiguously.)
-* For reading images multiple times after download. (At least two reads are
- needed, because the first pass only parses the headers.)
-* As a memory buffer for image coding processes to mmap. (For details, see
- [image.md](image.md).)
-
-Crucially, the cache *does not* understand Cache-Control headers, and will never
-skip a download when requested by a user. Similarly, loading a "cache:" URL
-(e.g. view source) is guaranteed to never make a network request.
-
-Future directions: for non-JS buffers, we could kill idle processes and reload
-them on-demand from the cache. This could solve the problem of spawning too many
-processes that then do nothing.
+* For rewinding buffers on incorrect charset guess. (In practice,
+ this is almost never used, because the first chunk we read tends to
+ determine the charset unambiguously.)
+* For reading images multiple times after download. (At least two reads
+ are needed, because the first pass only parses the headers.)
+* As a memory buffer for image coding processes to mmap. (For details,
+ see [image.md](image.md).)
+
+Crucially, the cache *does not* understand Cache-Control headers, and
+will never skip a download when requested by a user. Similarly, loading
+a "cache:" URL (e.g. view source) is guaranteed to never make a network
+request.
+
+Future directions: for non-JS buffers, we could kill idle processes and
+reload them on-demand from the cache. This could solve the problem of
+spawning too many processes that then do nothing.
## Parsing HTML
-The character decoder and the HTML parser are implementations of the WHATWG
-standard, and are available as [separate](https://git.sr.ht/~bptato/chagashi)
+The character decoder and the HTML parser are implementations of the
+WHATWG standards, and are available as
+[separate](https://git.sr.ht/~bptato/chagashi)
[libraries](https://git.sr.ht/~bptato/chame).
-The decoding and parsing of HTML documents happens in buffer processes. This
-operation is asynchronous; when bytes from the network are exhausted, the buffer
-will 1) partially render the current document as-is, 2) return it to the pager
-so that the user can interact with the document.
+Buffer processes decode and parse HTML documents asynchronously. When
+bytes from the network are exhausted, the buffer will 1) partially
+render the current document as-is, 2) return it to the pager so that the
+user can interact with the document.
Character encoding detection is rather primitive; the list specified in
-`encoding.document-charset` is enumerated until either no errors are produced by
-the decoder, or no more charsets exist. In some edge cases, the document must be
-(and is) re-downloaded from the cache, but this pretty much never happens in
-real-world scenarios. (The most common case is that the UTF-8 validator just
-runs through the entire document without reporting errors.)
+`encoding.document-charset` is enumerated until either no errors are
+produced by the decoder, or no more charsets exist. In some extremely
+rare edge cases, the document is re-downloaded from the cache, but this
+pretty much never happens. (The most common case is that the UTF-8
+validator just runs through the entire document without reporting
+errors.)
-The HTML parser then consumes the input buffer, which on the happy path (valid
-UTF-8) is just whatever we pulled from the network as-is. In some cases, a
-script calls document.write and then the parser is called re-entrantly.
-(Debugging this is not very fun.)
+The HTML parser then consumes the decoded (or validated) input buffer.
+In some cases, a script calls document.write and then the parser is
+called recursively. (Debugging this is not very fun.)
## JavaScript
-QuickJS is used by both the pager as a scripting language, and by buffers for
-running on-page scripts when JavaScript is enabled.
+QuickJS is used by both the pager as a scripting language, and by
+buffers for running on-page scripts when JavaScript is enabled.
The core JS related functionality has been separated out into the
-[Monoucha](https://git.sr.ht/~bptato/monoucha) library, so it can be used
-outside of Chawan too. Interested readers are invited to read the Monoucha
-[manual](https://git.sr.ht/~bptato/monoucha/tree/master/doc/manual.md) as well.
+[Monoucha](https://git.sr.ht/~bptato/monoucha) library, so it can be
+used outside of Chawan too. Interested readers are invited to read the
+[manual](https://git.sr.ht/~bptato/monoucha/tree/master/doc/manual.md).
### General
-To avoid having to type out all the type conversion & error handling code
-manually, we have JS pragmas to automagically turn Nim procedures into
-JavaScript functions. An explanation of what these pragmas are & what they do
-can be found in the header of js/javascript.nim.
+To avoid having to type out all the type conversion & error handling
+code manually, we have JS pragmas to automagically turn Nim procedures
+into JavaScript functions. An explanation of what these pragmas are &
+what they do can be found in the header of js/javascript.nim.
-The type conversion itself is handled by the overloaded toJS function and the
-generic fromJS function. toJS returns a JSValue, the native data type of
-QuickJS. fromJS returns a Result[T, JSError], which is interpreted as follows:
+The type conversion itself is handled by the overloaded toJS function
+and the generic fromJS function. toJS returns a JSValue, the native
+data type of QuickJS. fromJS returns a Result[T, JSError], which is
+interpreted as follows:
* ok(T) is successful conversion.
* err(JSError) is an error in the conversion.
* ok(nil) for reference types is null. For non-nullable types, null is
ok(none(T)).
-* err(nil) is JS_EXCEPTION, i.e. an exception has been thrown and is being
- propagated.
+* err(nil) is JS_EXCEPTION, i.e. an exception has been thrown and is
+ being propagated.
An additional point of interest is reference types: ref types registered
-with the registerType macro can be freely passed to JS, and the function-
-defining macros set functions on their JS prototypes. When a ref type is passed
-to JS, a shim JS object is associated with the Nim object, and will remain in
-memory until neither Nim nor JS has references to it.
-
-Effectively, this means that you can expose Nim objects to JS and take Nim
-objects as arguments through the automagical .jsfunc pragma (& friends) without
-having to bother with error-prone manual reference counting. How this is
-achieved is detailed below. (You generally don't need the following info
-unless you're debugging the JS type conversion logic, in which case I offer my
-condolences.)
+with the registerType macro can be freely passed to JS, and the
+function-defining macros set functions on their JS prototypes. When
+a ref type is passed to JS, a shim JS object is associated with the
+Nim object, and will remain in memory until neither Nim nor JS has
+references to it.
+
+This means that you can expose Nim objects to JS and take Nim objects
+as arguments through the .jsfunc pragma (& friends) without having
+to bother with manual reference counting. How this is achieved is
+detailed below. (TODO: this probably belongs in the Monoucha manual...)
In fact, there is a complication in this system: QuickJS has a reference-
counting GC, but Nim also has a reference-counting GC. Associating two objects
@@ -265,19 +274,21 @@ cycle collector can break up. A cross-GC cycle collector is obviously out of
question; then it would be easier to just replace the entire GC in one of the
runtimes.
-So instead, we hook into the QuickJS cycle collector (through a custom
-patch). Every time a JS companion object of a Nim object would be freed, we
-first check if the Nim object still has references from Nim, and if yes, prevent
-the JS object from being freed by "moving" a reference to the JS object
+So instead, we patch a hook into the QuickJS cycle collector. Every time
+a JS companion object of a Nim object would be freed, we first check if
+the Nim object still has references from Nim, and if yes, prevent the JS
+object from being freed by "moving" a reference to the JS object
(i.e. unref Nim, ref JS).
-Then, if we want to pass the object to JS again, we add no references to the JS
-object, only to the Nim object. By this, we "moved" the reference back to JS.
+Then, if we want to pass the object to JS again, we add no references to
+the JS object, only to the Nim object. By this, we "moved" the reference
+back to JS.
-This way, the Nim cycle collector can destroy the object without problems if no
-more references to it exist. But also, if you set some properties on the JS
-companion object, it will remain even if no more references exist to it in JS
-for some time, only in Nim. i.e. this works:
+This way, the Nim cycle collector can destroy the object without
+problems if no more references to it exist. But also, if you set some
+properties on the JS companion object, it will remain even if no more
+references exist to it in JS for some time, only in Nim. i.e. this
+works:
```js
document.querySelector("html").canary = "chirp";
@@ -286,81 +297,83 @@ console.log(document.querySelector("html").canary); /* chirp */
### JS in the pager
-Keybindings can be assigned JavaScript functions in the config, and then the
-pager executes those when the keybindings are pressed.
+Keybindings can be assigned JavaScript functions in the config, and
+then the pager executes those when the keybindings are pressed.
-Also, contents of the start.startup-script option are executed at startup. This
-is used when `cha` is called with the `-r` flag.
+Also, contents of the start.startup-script option are executed at
+startup. This is used when `cha` is called with the `-r` flag.
-There *is* an API, described at [api.md](api.md). Web APIs are exposed to
-pager too, but you cannot operate on the DOMs themselves from the pager, unless
-you create one yourself with DOMParser.parseFromString.
+There *is* an API, described at [api.md](api.md). Web APIs are exposed
+to pager too, but you cannot operate on the DOMs themselves from the
+pager, unless you create one yourself with DOMParser.parseFromString.
-[config.md](config.md) describes all commands that are used in the default
-config.
+[config.md](config.md) describes all commands that are used in the
+default config.
### JS in the buffer
-The DOM is implemented through the same wrappers as those in pager. (Obviously,
-the pager modules are not exposed to buffer JS.)
-
-Aside from document.write, it is mostly straightforward, and usually works OK,
-though too many things are missing to really make it useful.
+The DOM is implemented through the same wrappers as those in pager,
+except the pager modules are not exposed to buffer JS.
-As for document.write: don't ask. It works as far as I can tell, but I wouldn't
-know why.
+Aside from document.write, it is mostly straightforward, and usually
+works OK, though too many things are missing to really make it useful.
-## Styling
+As for document.write: don't ask. It works as far as I can tell, but
+I wouldn't know why.
-css/ contains everything related to styling: CSS parsing and cascading.
+## Styling, layout
-The parser is not very interesting, it's just an implementation of the CSS 3
-parsing module. The latest iteration of the selector parser is pretty good. The
-media query parser and the CSS value parser both work OK, but are missing
-some commonly used features like variables.
+css/ contains CSS parsing, cascading, layout, and rendering.
-Cascading is slow, though it could be slower. Chawan has style caching, so
-re-styles are normally very fast. Also, a hash map is used for reducing initial
-style calculation times. However, we don't have a Bloom filter yet.
+Note that CSS (at least 2.0 and onward) was designed for pixel-based
+displays, not for character-based ones. So we have to round a lot,
+and sometimes this goes wrong. (This is mostly solved by the omission of
+certain problematic properties and some heuristics in the layout engine.)
-## Layout
+Also, some (now) commonly used features like CSS grid are not
+implemented yet, so websites using those look ugly.
-Layout can be found in the layout/ module.
+### Parsing, cascading
-It has some problems:
+The parser is not very interesting, it's just an implementation of the
+CSS 3 parsing module. The latest iteration of the selector parser is
+pretty good. The media query parser and the CSS value parser both work
+OK, but are missing some commonly used features like variables.
-* CSS was designed for pixel-based displays, not for character-based ones. So we
- have to round a lot, and sometimes this goes wrong. (This is mostly solved by
- some basic heuristics inside the layout engine.)
-* Some (now) commonly used features like grid are not implemented, so websites
- using those look ugly.
-* It's slow on large documents, because we don't have partial layouting
- capabilities.
+Cascading is slow, though it could be slower. Chawan caches the style
+tree, so re-styles are normally very fast. Also, a hash map is used for
+reducing initial style calculation times. However, we don't have a Bloom
+filter yet.
-Our layout engine is a rather simple procedural layout implementation. It runs
-in two passes.
+### Layout
-In the first pass, it generates the layout tree; this is important because rules
-for generating anonymous boxes are surprisingly involved. (Specifically,
-anonymous inline box handling is kind of a mess.)
+Our layout engine is a rather simple procedural layout implementation.
+It runs in two passes.
-The second pass then does the actual arrangement of the boxes on the screen. The
-output tree uses relative coordinates; that is, every box is positioned relative
-to its parent.
+* Build a layout tree. Anonymous boxes are generated here. After this
+ pass, the tree is no longer mutated, only the `state` and `render`
+ fields of the respective boxes.
+* Position said boxes, always relative to their parent. This pass
+ sets the values in the `state` field.
-Layout is fully recursive. This means that after a certain nesting depth, the
-buffer will run out of stack space and promptly crash.
+Layout is fully recursive. This means that after a certain nesting
+depth, the buffer will run out of stack space and promptly crash.
-Since we do not cache layout results, and the whole page is layouted (no partial
-layouting), it gets quite slow on large documents.
+Since we do not cache layout results, and the whole page is layouted,
+it gets quite slow on large documents. (Layout is being incrementally
+refactored to make implementing a cache simpler.)
### Rendering
-After layout is finished, the document is rendered onto a text-based canvas,
-which is represented as a sequence of strings associated with their formatting.
+After layout is finished, the document is rendered onto a text-based
+canvas, which is represented as a sequence of strings associated with
+their formatting.
-Again, the entire document is rendered, which is the main reason why Chawan
-performs poorly on large documents.
+The entire document is rendered, and this is our main performance
+bottleneck right now. (In fact, rendering takes much longer than
+layout. Styling is even slower, but that's less of a problem because
+it's cached.)
-The positive side of this is that search is very simple (and fast), since we are
-just running regexes over a linear sequence of strings.
+The positive side of this design is that search is very simple (and
+fast), since we are just running regexes over a linear sequence of
+strings.