path: root/bash/talk-to-computer/corpus/programming/lil_guide.md

# Multi-paradigm Programming with Lil - A Guide to Lil's Diverse Styles

## Introduction

Lil is a richly multi-paradigm scripting language designed for the Decker creative environment. It seamlessly blends concepts from **imperative**, **functional**, **declarative**, and **vector-oriented** programming languages. This flexibility allows developers to choose the most effective and ergonomic approach for a given task, whether it's managing application state, manipulating complex data structures, or performing efficient bulk computations. Understanding these paradigms is key to writing elegant, efficient, and idiomatic Lil code.

## Core Concepts

Lil's power comes from the way it integrates four distinct programming styles.

### Imperative Programming

This is the traditional, statement-by-statement style of programming. It involves creating variables, assigning values to them, and using loops and conditionals to control the flow of execution.

  - **Assignment:** The colon (`:`) is used for assignment.
  - **Control Flow:** Lil provides `if`/`elseif`/`else` for conditionals and `while` and `each` for loops.
  - **State Management:** State is typically managed by assigning and re-assigning values to variables, often stored in the properties of Decker widgets between event handlers.

<!-- end list -->

```lil
# Imperative approach to summing a list
total: 0
numbers: [10, 20, 30]
each n in numbers do
  total: total + n
end
# total is now 60
```

### Functional Programming

The functional style emphasizes pure functions, immutability, and the composition of functions without side-effects.

  - **Immutability:** All core data structures (lists, dictionaries, tables) have copy-on-write semantics. Modifying one does not alter the original value but instead returns a new, amended value.
  - **First-Class Functions:** Functions are values that can be defined with `on`, assigned to variables, and passed as arguments to other functions.
  - **Expressions over Statements:** Every statement in Lil is an expression that returns a value. An `if` block returns the value of its executed branch, and an `each` loop returns a new collection containing the result of each iteration.

<!-- end list -->

```lil
# Functional approach using a higher-order function
on twice f x do
  f[f[x]]
end

on double x do
  x * 2
end

result: twice[double 10] # result is 40
```

### Declarative (Query-based) Programming

For data manipulation, Lil provides a powerful declarative query engine that resembles SQL. Instead of describing *how* to loop through and filter data, you declare *what* data you want.

  - **Queries:** Use `select`, `update`, and `extract` to query tables (and other collection types).
  - **Clauses:** Filter, group, and sort data with `where`, `by`, and `orderby` clauses.
  - **Readability:** Queries often result in more concise and readable code for data transformation tasks compared to imperative loops.

<!-- end list -->

```lil
# Declarative query to find developers
people: insert name age job with
 "Alice"  25 "Development"
 "Sam"    28 "Sales"
 "Thomas" 40 "Development"
end

devs: select name from people where job="Development"
# devs is now a table with the names "Alice" and "Thomas"
```

### Vector-Oriented Programming

Influenced by languages like APL and K, this paradigm focuses on applying operations to entire arrays or lists (vectors) at once, a concept known as **conforming**.

  - **Conforming Operators:** Standard arithmetic operators (`+`, `-`, `*`, `/`) work element-wise on lists.
  - **Efficiency:** Vector operations are significantly more performant than writing equivalent imperative loops.
  - **The `@` Operator:** The "apply" operator (`@`) can be used to apply a function to each element of a list or to select multiple elements from a list by index.

<!-- end list -->

```lil
# Vector-oriented approach to add 10 to each number
numbers: [10, 20, 30]
result: numbers + 10 # result is [20, 30, 40]
```

-----

## Key Principles

  - **Right-to-Left Evaluation:** Expressions are evaluated from right to left unless overridden by parentheses `()`. This is a fundamental rule that affects how all expressions are composed.
  - **Copy-on-Write Immutability:** Lists, Dictionaries, and Tables are immutable. Operations like `update` or indexed assignments on an expression `(foo)[1]:44` return a new value, leaving the original unchanged. Direct assignment `foo[1]:44` is required to modify the variable `foo` itself.
  - **Data-Centric Design:** The language provides powerful, built-in tools for data manipulation, especially through its query engine and vector operations.
  - **Lexical Scoping:** Variables are resolved based on their location in the code's structure. Functions "close over" variables from their containing scope, enabling patterns like counters and encapsulated state.

-----

## Implementation/Usage

The true power of Lil emerges when you mix these paradigms to solve problems cleanly and efficiently.

### Basic Example

Here, we combine an imperative loop with a vector-oriented operation to process a list of lists.

```lil
# Calculate the magnitude of several 2D vectors
vectors: [[3,4], [5,12], [8,15]]
magnitudes: []

# Imperative loop over the list of vectors
each v in vectors do
  # mag is a vector-oriented unary operator
  magnitudes: magnitudes & [mag v]
end

# magnitudes is now [5, 13, 17]
```

### Advanced Example

This example defines a functional-style utility function (`avg`) and uses it within a declarative query to summarize data, an approach common in data analysis.

```lil
# Functional helper function
on avg x do
  (sum x) / count x
end

# A table of sales data
sales: insert product category price with
 "Apple"  "Fruit"  0.5
 "Banana" "Fruit"  0.4
 "Bread"  "Grain"  2.5
 "Rice"   "Grain"  3.0
end

# Declarative query that uses the functional helper
avgPriceByCategory: select category:first category avg_price:avg[price] by category from sales

# avgPriceByCategory is now:
# +----------+-----------+
# | category | avg_price |
# +----------+-----------+
# | "Fruit"  | 0.45      |
# | "Grain"  | 2.75      |
# +----------+-----------+
```

-----

## Common Patterns

### Pattern 1: Query over Loop

Instead of manually iterating with `each` to filter or transform a collection, use a declarative `select` or `extract` query. This is more concise, often faster, and less error-prone.

```lil
# Instead of this imperative loop...
high_scores: []
scores: [88, 95, 72, 100, 91]
each s in scores do
  if s > 90 then
    high_scores: high_scores & [s]
  end
end

# ...use a declarative query.
high_scores: extract value where value > 90 from scores
# high_scores is now [95, 100, 91]
```

### Pattern 2: Function Application with `@`

For simple element-wise transformations on a list, using the `@` operator with a function is cleaner than writing an `each` loop.

```lil
# Instead of this...
names: ["alice", "bob", "charlie"]
capitalized: []
on capitalize s do first s & (1 drop s) end # Simple capitalize, for demo
each n in names do
  capitalized: capitalized & [capitalize n]
end

# ...use the more functional and concise @ operator.
on capitalize s do first s & (1 drop s) end
capitalized: capitalize @ names
# capitalized is now ["Alice", "Bob", "Charlie"]
```

-----

## Best Practices

  - **Embrace Queries:** For any non-trivial data filtering, grouping, or transformation, reach for the query engine first.
  - **Use Vector Operations:** When performing arithmetic or logical operations on lists, use conforming operators (`+`, `<`, `=`) instead of loops for better performance and clarity.
  - **Distinguish Equality:** Use the conforming equals `=` within query expressions. Use the non-conforming match `~` in `if` or `while` conditions to avoid accidentally getting a list result.
  - **Encapsulate with Functions:** Use functions to create reusable components and manage scope, especially for complex logic within Decker event handlers.

-----

## Common Pitfalls

  - **Right-to-Left Confusion:** Forgetting that `3*2+5` evaluates to `21`, not `11`. Use parentheses `(3*2)+5` to enforce the desired order of operations.
  - **Expecting Mutation:** Believing that `update ... from my_table` changes `my_table`. It returns a *new* table. You must reassign it: `my_table: update ... from my_table`.
  - **Comma as Argument Separator:** Writing `myfunc[arg1, arg2]`. This creates a list of two items and passes it as a single argument. The correct syntax is `myfunc[arg1 arg2]`.
  - **Using `=` in `if`:** Writing `if some_list = some_value` can produce a list of `0`s and `1`s. An empty list `()` is falsey, but a list like `[0,0]` is truthy. Use `~` for a single boolean result in control flow.

-----

## Performance Considerations

Vector-oriented algorithms are significantly faster and more memory-efficient than their imperative, element-by-element counterparts. The Lil interpreter is optimized for these bulk operations. For example, replacing values in a list using a calculated mask is preferable to an `each` loop with a conditional inside.

```lil
# Slow, iterative approach
x: [1, 10, 2, 20, 3, 30]
result: each v in x
  if v < 5 99 else v end
end

# Fast, vector-oriented approach
mask: x < 5                 # results in [1,0,1,0,1,0]
result: (99 * mask) + (x * !mask)
```

-----

## Integration Points

The primary integration point for Lil is **Decker**. Lil scripts are attached to Decker widgets, cards, and the deck itself to respond to user events (`on click`, `on keydown`, etc.). All paradigms are useful within Decker:

  - **Imperative:** To sequence actions, like showing a dialog and then navigating to another card.
  - **Declarative:** To query data stored in a `grid` widget or to find specific cards in the deck, e.g., `extract value where value..widgets.visited.value from deck.cards`.
  - **Functional/Vector:** To process data before displaying it, without needing slow loops.

-----

## Troubleshooting

### Problem 1: An `if` statement behaves unpredictably with list comparisons.

  - **Symptoms:** An `if` block either never runs or always runs when comparing a value against a list.
  - **Solution:** You are likely using the conforming equals operator (`=`), which returns a list of boolean results. In a conditional, you almost always want the non-conforming match operator (`~`), which returns a single `1` or `0`.

### Problem 2: A recursive function crashes with a stack overflow on large inputs.

  - **Symptoms:** The script terminates unexpectedly when a recursive function is called with a large number or deep data structure.
  - **Solution:** Lil supports **tail-call elimination**. Ensure your recursive call is the very last operation performed in the function. If it's part of a larger expression (e.g., `1 + my_func[...]`), it is not in a tail position. Rewrite the function to accumulate its result in an argument.

-----

## Examples in Context

**Use Case: A Simple To-Do List in Decker**

Imagine a Decker card with a `grid` widget named "tasks" (with columns "desc" and "done") and a `field` widget named "summary".

```lil
# In the script of the "tasks" grid, to update when it's changed:
on change do
  # Use a DECLARATIVE query to get the done/total counts.
  # The query source is "me.value", the table in the grid.
  stats: extract done:sum done total:count done from me.value

  # Use IMPERATIVE assignment to update the summary field.
  summary.text: format["%d of %d tasks complete.", stats.done, stats.total]
end
```

This tiny script uses a declarative query to read the state and an imperative command to update the UI, demonstrating a practical mix of paradigms.