diff options
Diffstat (limited to 'js/scripting-lang/README.md')
| -rw-r--r-- | js/scripting-lang/README.md | 430 |
1 files changed, 252 insertions, 178 deletions
diff --git a/js/scripting-lang/README.md b/js/scripting-lang/README.md index 73ccbf1..cc650de 100644 --- a/js/scripting-lang/README.md +++ b/js/scripting-lang/README.md @@ -1,220 +1,294 @@ -# Simple Scripting Language +# Scripting Language: A Combinator-Based Functional Language -A functional programming language with immutable variables, first-class functions, and pattern matching. +A functional programming language built on a **combinator foundation** that eliminates parsing ambiguity while preserving intuitive syntax. Every operation is a function call under the hood, creating a consistent and extensible language architecture. -## Features +## Why This Approach? -- **Immutable Variables**: Variables cannot be reassigned -- **First-Class Functions**: Functions can be passed as arguments and stored in data structures -- **Lexical Scoping**: Functions create their own scope -- **Pattern Matching**: Case expressions with wildcard support -- **Table Literals**: Lua-style tables with both array-like and key-value entries -- **Standard Library**: Built-in higher-order functions (`map`, `compose`, `pipe`, `apply`, `filter`, `reduce`, `fold`, `curry`) -- **IO Operations**: Built-in input/output operations (`..in`, `..out`, `..assert`) -- **Floating Point Arithmetic**: Full support for decimal numbers -- **Unary Minus**: Support for negative numbers (e.g., `-1`, `-3.14`) +### The Problem We Solved +Traditional language parsers struggle with **operator precedence ambiguity**. Consider `f x + y` - should this be `(f x) + y` or `f (x + y)`? Most languages solve this with complex precedence tables, but we took a different approach. -## Syntax +### The Combinator Solution +We translate **every operation into a function call**: +- `x + y` becomes `add(x, y)` +- `x - y` becomes `subtract(x, y)` +- `f x` becomes `apply(f, x)` +- `true and false` becomes `logicalAnd(true, false)` -### Basic Operations -``` -/* Arithmetic */ -x : 5 + 3; -y : 10 - 2; -z : 4 * 3; -w : 15 / 3; -neg : -5; /* Unary minus */ - -/* Comparisons */ -result : x > y; -equal : a = b; -not_equal : a != b; - -/* Logical */ -and_result : true and false; -or_result : true or false; -``` +This eliminates ambiguity entirely while keeping the syntax clean and intuitive. -### Variables and Functions -``` -/* Immutable variables */ -x : 42; -y : "hello"; +### Benefits +- **Zero Ambiguity**: Every expression has exactly one interpretation +- **Functional Foundation**: Everything is a function, enabling powerful abstractions +- **Extensible**: Add new operations by simply adding combinator functions +- **Consistent**: All operations follow the same pattern +- **Preserves Syntax**: Existing code continues to work unchanged -/* Function definition */ -f : x -> x * 2; +## How It Works -/* Function call */ -result : f 5; +### Architecture Overview ``` - -### Tables +Source Code → Lexer → Parser → AST → Interpreter → Result + ↓ ↓ ↓ ↓ + Tokens → Combinator Translation → Function Calls ``` -/* Table literal */ -table : {1, 2, 3, key: "value"}; -/* Table access */ -first : table[1]; -value : table.key; -nested : table.key.subkey; +### The Magic: Operator Translation +When you write `x + y * z`, the parser automatically translates it to: +```javascript +add(x, multiply(y, z)) ``` -### Pattern Matching -``` -/* Case expression */ -result : case x of - 1 : "one" - 2 : "two" - _ : "other"; +This happens transparently - you write natural syntax, but get functional semantics. + +### Function Application with Juxtaposition +Functions are applied by placing arguments next to them: +```javascript +f : x -> x * 2; +result : f 5; // apply(f, 5) → 10 ``` -### IO Operations +### Function Composition with `via` +Compose functions naturally: +```javascript +f : x -> x * 2; +g : x -> x + 1; +result : f via g 5; // compose(f, g)(5) → 12 ``` -/* Output */ -..out "Hello, World!"; -/* Input */ -name : ..in; +## Language Features -/* Assertion */ -..assert x = 5; -``` +### Core Philosophy +- **Immutable by Default**: Variables cannot be reassigned +- **Functions First**: Everything is a function or function application +- **Pattern Matching**: Natural case expressions with wildcards +- **Lexical Scoping**: Functions create their own scope -### Standard Library +### Key Features +- **Combinator Foundation**: All operations are function calls +- **Function Composition**: `via` operator for natural composition +- **Pattern Matching**: `when` expressions with wildcard support +- **Tables**: Lua-style tables with array and key-value support +- **Standard Library**: Higher-order functions (`map`, `compose`, `pipe`, etc.) +- **IO Operations**: Built-in input/output (`..in`, `..out`, `..assert`) + +## Development Workflow + +### Testing Strategy +We use a **progressive testing approach** to ensure quality: + +#### 1. Scratch Tests (`scratch_tests/`) +**Purpose**: Rapid prototyping and debugging +- **Location**: `scratch_tests/*.txt` +- **Use Case**: Isolating specific issues, testing new features +- **Naming**: `test_<feature>_<purpose>.txt` (e.g., `test_precedence_simple.txt`) +- **Lifecycle**: Temporary, can be deleted after issue resolution + +#### 2. Unit Tests (`tests/`) +**Purpose**: Comprehensive feature coverage +- **Location**: `tests/*.txt` +- **Use Case**: Validating complete feature implementations +- **Naming**: `##_<feature_name>.txt` (e.g., `01_lexer_basic.txt`) +- **Lifecycle**: Permanent, part of regression testing + +#### 3. Integration Tests (`tests/`) +**Purpose**: Testing feature combinations +- **Location**: `tests/integration_*.txt` +- **Use Case**: Ensuring features work together +- **Naming**: `integration_##_<description>.txt` + +### Development Process + +#### When Adding New Features +1. **Create scratch test** to explore the feature +2. **Implement incrementally** with frequent testing +3. **Debug with `DEBUG=1`** for detailed output +4. **Promote to unit test** when feature is stable +5. **Add integration tests** for feature combinations + +#### When Fixing Bugs +1. **Create minimal scratch test** reproducing the issue +2. **Debug with `DEBUG=1`** to understand the problem +3. **Fix the issue** and verify with scratch test +4. **Update existing tests** if needed +5. **Clean up scratch tests** after resolution + +#### When Promoting Scratch Tests +```bash +# Test the scratch test +bun run lang.js scratch_tests/test_new_feature.txt + +# If it passes, promote to unit test +cp scratch_tests/test_new_feature.txt tests/16_new_feature.txt + +# Update test numbering if needed +# Clean up scratch test +rm scratch_tests/test_new_feature.txt ``` -/* Map */ -double : x -> x * 2; -squared : map @double 5; -/* Filter */ -isPositive : x -> x > 0; -filtered : filter @isPositive 5; +### Debugging Tools -/* Compose */ -f : x -> x + 1; -g : x -> x * 2; -h : compose @f @g; -result : h 5; /* (5 * 2) + 1 = 11 */ +#### Enable Debug Mode +```bash +DEBUG=1 bun run lang.js script.txt ``` -## Usage +This shows: +- Token stream from lexer +- AST structure from parser +- Function call traces +- Scope information -### Running Scripts +#### Call Stack Tracking +The interpreter tracks function calls to detect infinite recursion: ```bash -node lang.js script.txt +# Shows call statistics after execution +bun run lang.js script.txt ``` -### Testing -The project uses a structured testing approach with unit and integration tests: - -#### Unit Tests -Located in `tests/` directory, each focusing on a specific language feature: -- `01_lexer_basic.txt` - Basic lexer functionality -- `02_arithmetic_operations.txt` - Arithmetic operations -- `03_comparison_operators.txt` - Comparison operators -- `04_logical_operators.txt` - Logical operators -- `05_io_operations.txt` - IO operations -- `06_function_definitions.txt` - Function definitions -- `07_case_expressions.txt` - Case expressions and pattern matching -- `08_first_class_functions.txt` - First-class function features -- `09_tables.txt` - Table literals and access -- `10_standard_library.txt` - Standard library functions - -#### Integration Tests -Test combinations of multiple features: -- `integration_01_basic_features.txt` - Basic feature combinations -- `integration_02_pattern_matching.txt` - Pattern matching with other features -- `integration_03_functional_programming.txt` - Functional programming patterns - -#### Running Tests +### Running Tests ```bash # Run all tests ./run_tests.sh -# Run individual tests -node lang.js tests/01_lexer_basic.txt -node lang.js tests/integration_01_basic_features.txt +# Run individual test +bun run lang.js tests/01_lexer_basic.txt + +# Run scratch test +bun run lang.js scratch_tests/test_debug_issue.txt ``` -## Implementation Details +## Current Status -### Architecture -- **Lexer**: Tokenizes input into tokens (numbers, identifiers, operators, etc.) -- **Parser**: Builds Abstract Syntax Tree (AST) from tokens -- **Interpreter**: Executes AST with scope management - -### Key Components -- **Token Types**: Supports all basic operators, literals, and special tokens -- **AST Nodes**: Expression, statement, and declaration nodes -- **Scope Management**: Lexical scoping with proper variable resolution -- **Error Handling**: Comprehensive error reporting for parsing and execution - -## Recent Fixes - -### ✅ Parser Ambiguity with Unary Minus Arguments (Latest Fix) -- **Issue**: `filter @isPositive -3` was incorrectly parsed as binary operation instead of function call with unary minus argument -- **Root Cause**: Parser treating `FunctionReference MINUS` as binary minus operation -- **Solution**: Added special case in `parseExpression()` to handle `FunctionReference MINUS` pattern -- **Status**: ✅ Resolved - Standard library functions now work with negative arguments - -### ✅ Unary Minus Operator -- **Issue**: Stack overflow when parsing negative numbers (e.g., `-1`) -- **Root Cause**: Parser lacked specific handling for unary minus operator -- **Solution**: Added `UnaryMinusExpression` parsing and evaluation -- **Status**: ✅ Resolved - All tests passing - -### ✅ IO Operation Parsing -- **Issue**: IO operations not parsed correctly at top level -- **Solution**: Moved IO parsing to proper precedence level -- **Status**: ✅ Resolved - -### ✅ Decimal Number Support -- **Issue**: Decimal numbers not handled correctly -- **Solution**: Updated lexer and interpreter to use `parseFloat()` -- **Status**: ✅ Resolved - -## Known Issues - -### 🔄 Logical Operator Precedence -- **Issue**: Logical operators (`and`, `or`, `xor`) have incorrect precedence relative to function calls -- **Example**: `isEven 10 and isPositive 5` is parsed as `isEven(10 and isPositive(5))` instead of `(isEven 10) and (isPositive 5)` -- **Impact**: Complex expressions with logical operators may not evaluate correctly -- **Status**: 🔄 In Progress - Working on proper operator precedence hierarchy -- **Workaround**: Use parentheses to explicitly group expressions: `(isEven 10) and (isPositive 5)` - -### 🔄 Parentheses Parsing with Logical Operators -- **Issue**: Some expressions with logical operators inside parentheses fail to parse -- **Example**: `add (multiply 3 4) (isEven 10 and isPositive 5)` may fail with parsing errors -- **Status**: 🔄 In Progress - Related to logical operator precedence issue - -## Development - -### File Structure -``` -. -├── lang.js # Main implementation -├── test.txt # Comprehensive test file -├── tests/ # Unit and integration tests -│ ├── 01_lexer_basic.txt -│ ├── 02_arithmetic_operations.txt -│ ├── ... -│ ├── integration_01_basic_features.txt -│ ├── integration_02_pattern_matching.txt -│ └── integration_03_functional_programming.txt -├── run_tests.sh # Test runner script -├── FIXME.md # Issues and fixes documentation -└── README.md # This file +### ✅ Recently Completed +- **Function Composition**: `via` operator and `@` function references +- **Enhanced Standard Library**: `compose` and `pipe` with partial application +- **Combinator Foundation**: All operators translate to function calls + +### 🔧 Active Issues +- **Priority 1**: Precedence issues (binary minus operator) +- **Priority 2**: Test suite failures (blocked by precedence fix) + +### 📋 Implementation Plans +See `design/implementation/` for detailed plans: +- `PRECEDENCE_RESOLUTION_PLAN.md` - Active precedence fix +- `FUNCTION_COMPOSITION_PLAN.md` - Completed composition features + +## Quick Start + +### Installation +```bash +# Clone the repository +git clone <repository-url> +cd scripting-lang + +# Install dependencies (if any) +npm install +# or +bun install ``` -### Debugging -Enable debug mode by setting `DEBUG=true`: +### Running Scripts ```bash -DEBUG=true node lang.js script.txt +# Basic script execution +bun run lang.js script.txt + +# With debug output +DEBUG=1 bun run lang.js script.txt +``` + +### Example Script +```javascript +/* Basic arithmetic with combinator translation */ +x : 5; +y : 3; +result : x + y; // becomes add(x, y) internally + +/* Function definition and application */ +double : x -> x * 2; +result : double 5; // becomes apply(double, 5) internally + +/* Function composition */ +add1 : x -> x + 1; +result : double via add1 5; // becomes compose(double, add1)(5) internally + +/* Pattern matching */ +message : when result is + 10 then "ten" + 12 then "twelve" + _ then "other"; + +/* Output */ +..out message; +``` + +## Architecture Deep Dive + +### Combinator Foundation +Every operation is implemented as a function call to standard library combinators: + +```javascript +// Arithmetic +x + y → add(x, y) +x - y → subtract(x, y) +x * y → multiply(x, y) + +// Comparison +x = y → equals(x, y) +x > y → greaterThan(x, y) + +// Logical +x and y → logicalAnd(x, y) +not x → logicalNot(x) + +// Function application +f x → apply(f, x) ``` +### Standard Library Combinators +The language includes a comprehensive set of combinators: +- **Arithmetic**: `add`, `subtract`, `multiply`, `divide`, `negate` +- **Comparison**: `equals`, `greaterThan`, `lessThan`, etc. +- **Logical**: `logicalAnd`, `logicalOr`, `logicalNot` +- **Higher-Order**: `map`, `compose`, `pipe`, `apply`, `filter` + +### Parser Architecture +The parser uses a **precedence climbing** approach with combinator translation: +1. **Lexer**: Converts source to tokens +2. **Parser**: Builds AST with operator-to-function translation +3. **Interpreter**: Evaluates AST using combinator functions + ## Contributing -1. Create focused unit tests for new features -2. Add integration tests for feature combinations -3. Update documentation -4. Run the full test suite before submitting changes \ No newline at end of file +### Development Guidelines +1. **Follow the combinator approach** for new operations +2. **Use scratch tests** for rapid prototyping +3. **Promote to unit tests** when features are stable +4. **Maintain backward compatibility** - existing code must work +5. **Document changes** in design documents + +### Code Style +- **Functional approach**: Prefer pure functions +- **Combinator translation**: All operations become function calls +- **Clear naming**: Descriptive function and variable names +- **Comprehensive testing**: Test edge cases and combinations + +### Testing Requirements +- **Unit tests** for all new features +- **Integration tests** for feature combinations +- **Backward compatibility** tests for existing code +- **Edge case coverage** for robust implementation + +## Documentation + +### Design Documents +- `design/PROJECT_ROADMAP.md` - Current status and next steps +- `design/COMBINATORS.md` - Combinator foundation explanation +- `design/implementation/` - Detailed implementation plans + +### Architecture +- **Combinator Foundation**: Eliminates parsing ambiguity +- **Functional Semantics**: Everything is a function +- **Extensible Design**: Easy to add new operations +- **Consistent Patterns**: All operations follow the same structure + +This language demonstrates how **functional programming principles** can solve real parsing problems while maintaining intuitive syntax. The combinator foundation provides a solid base for building powerful abstractions. \ No newline at end of file |