diff options
Diffstat (limited to 'js/scripting-lang/README.md')
| -rw-r--r-- | js/scripting-lang/README.md | 412 |
1 files changed, 244 insertions, 168 deletions
diff --git a/js/scripting-lang/README.md b/js/scripting-lang/README.md index 73ccbf1..b5d88d7 100644 --- a/js/scripting-lang/README.md +++ b/js/scripting-lang/README.md @@ -1,220 +1,296 @@ -# Simple Scripting Language +# Scripting Language: A Combinator-Based Functional Language -A functional programming language with immutable variables, first-class functions, and pattern matching. +A functional programming language built on a **combinator foundation** that eliminates parsing ambiguity while preserving intuitive syntax. Every operation is a function call under the hood, creating a consistent and extensible language architecture. -## Features +## Quick Start -- **Immutable Variables**: Variables cannot be reassigned -- **First-Class Functions**: Functions can be passed as arguments and stored in data structures -- **Lexical Scoping**: Functions create their own scope -- **Pattern Matching**: Case expressions with wildcard support -- **Table Literals**: Lua-style tables with both array-like and key-value entries -- **Standard Library**: Built-in higher-order functions (`map`, `compose`, `pipe`, `apply`, `filter`, `reduce`, `fold`, `curry`) -- **IO Operations**: Built-in input/output operations (`..in`, `..out`, `..assert`) -- **Floating Point Arithmetic**: Full support for decimal numbers -- **Unary Minus**: Support for negative numbers (e.g., `-1`, `-3.14`) - -## Syntax +### Running Scripts +```bash +# Basic script execution +bun run lang.js script.txt -### Basic Operations -``` -/* Arithmetic */ -x : 5 + 3; -y : 10 - 2; -z : 4 * 3; -w : 15 / 3; -neg : -5; /* Unary minus */ - -/* Comparisons */ -result : x > y; -equal : a = b; -not_equal : a != b; - -/* Logical */ -and_result : true and false; -or_result : true or false; +# With debug output +DEBUG=1 bun run lang.js script.txt ``` -### Variables and Functions -``` -/* Immutable variables */ -x : 42; -y : "hello"; +### Example Script +```javascript +/* Basic arithmetic with combinator translation */ +x : 5; +y : 3; +result : x + y; // becomes add(x, y) internally -/* Function definition */ -f : x -> x * 2; +/* Function definition and application */ +double : x -> x * 2; +result : double 5; // becomes apply(double, 5) internally -/* Function call */ -result : f 5; -``` +/* Function composition with @ operator */ +add1 : x -> x + 1; +ref : @double; // function reference +result : ref 5; // call referenced function -### Tables -``` -/* Table literal */ -table : {1, 2, 3, key: "value"}; +/* Pattern matching */ +message : when result is + 10 then "ten" + 12 then "twelve" + _ then "other"; -/* Table access */ -first : table[1]; -value : table.key; -nested : table.key.subkey; +/* Output */ +..out message; ``` -### Pattern Matching +## Current Status + +### ✅ Recently Completed +- **@ Operator**: Function reference syntax working perfectly +- **Standard Library**: All higher-order functions working with @ syntax +- **Precedence Issues**: All operator precedence issues resolved +- **Partial Application**: Fixed `reduce`, `fold`, `curry` functions + +### 🔧 Active Issues +- **Case Expression Parsing**: Fix "Unexpected token in parsePrimary: THEN" errors +- **Test Suite**: Get all tests passing (currently 8/18) + +### 📊 Test Results +- **Passing**: 8/18 tests (all core features working) +- **Failing**: 10/18 tests (due to case expression parsing) +- **Architecture**: Working correctly +- **Function Composition**: ✅ Complete and working + +## Why This Approach? + +### The Problem We Solved +Traditional language parsers struggle with **operator precedence ambiguity**. Consider `f x + y` - should this be `(f x) + y` or `f (x + y)`? Most languages solve this with complex precedence tables, but we took a different approach. + +### The Combinator Solution +We translate **every operation into a function call**: +- `x + y` becomes `add(x, y)` +- `x - y` becomes `subtract(x, y)` +- `f x` becomes `apply(f, x)` +- `true and false` becomes `logicalAnd(true, false)` + +This eliminates ambiguity entirely while keeping the syntax clean and intuitive. + +### Benefits +- **Zero Ambiguity**: Every expression has exactly one interpretation +- **Functional Foundation**: Everything is a function, enabling powerful abstractions +- **Extensible**: Add new operations by simply adding combinator functions +- **Consistent**: All operations follow the same pattern +- **Preserves Syntax**: Existing code continues to work unchanged + +## How It Works + +### Architecture Overview ``` -/* Case expression */ -result : case x of - 1 : "one" - 2 : "two" - _ : "other"; +Source Code → Lexer → Parser → AST → Interpreter → Result + ↓ ↓ ↓ ↓ + Tokens → Combinator Translation → Function Calls ``` -### IO Operations +### The Magic: Operator Translation +When you write `x + y * z`, the parser automatically translates it to: +```javascript +add(x, multiply(y, z)) ``` -/* Output */ -..out "Hello, World!"; -/* Input */ -name : ..in; +This happens transparently - you write natural syntax, but get functional semantics. -/* Assertion */ -..assert x = 5; +### Function Application with Juxtaposition +Functions are applied by placing arguments next to them: +```javascript +f : x -> x * 2; +result : f 5; // apply(f, 5) → 10 ``` -### Standard Library +### Function References with @ Operator +Reference functions without calling them: +```javascript +double_func : x -> x * 2; +ref : @double_func; // function reference +result : ref 5; // call referenced function ``` -/* Map */ -double : x -> x * 2; -squared : map @double 5; -/* Filter */ -isPositive : x -> x > 0; -filtered : filter @isPositive 5; +## Language Features + +### Core Philosophy +- **Immutable by Default**: Variables cannot be reassigned +- **Functions First**: Everything is a function or function application +- **Pattern Matching**: Natural case expressions with wildcards +- **Lexical Scoping**: Functions create their own scope -/* Compose */ -f : x -> x + 1; -g : x -> x * 2; -h : compose @f @g; -result : h 5; /* (5 * 2) + 1 = 11 */ +### Key Features +- **Combinator Foundation**: All operations are function calls +- **Function References**: `@` operator for function references +- **Pattern Matching**: `when` expressions with wildcard support +- **Tables**: Lua-style tables with array and key-value support +- **Standard Library**: Higher-order functions (`map`, `compose`, `pipe`, etc.) +- **IO Operations**: Built-in input/output (`..in`, `..out`, `..assert`) + +## Development Workflow + +### Testing Strategy +We use a **progressive testing approach** to ensure quality: + +#### 1. Scratch Tests (`scratch_tests/`) +**Purpose**: Rapid prototyping and debugging +- **Location**: `scratch_tests/*.txt` +- **Use Case**: Isolating specific issues, testing new features +- **Naming**: `test_<feature>_<purpose>.txt` (e.g., `test_precedence_simple.txt`) +- **Lifecycle**: Temporary, can be deleted after issue resolution + +#### 2. Unit Tests (`tests/`) +**Purpose**: Comprehensive feature coverage +- **Location**: `tests/*.txt` +- **Use Case**: Validating complete feature implementations +- **Naming**: `##_<feature_name>.txt` (e.g., `01_lexer_basic.txt`) +- **Lifecycle**: Permanent, part of regression testing + +#### 3. Integration Tests (`tests/`) +**Purpose**: Testing feature combinations +- **Location**: `tests/integration_*.txt` +- **Use Case**: Ensuring features work together +- **Naming**: `integration_##_<description>.txt` + +### Development Process + +#### When Adding New Features +1. **Create scratch test** to explore the feature +2. **Implement incrementally** with frequent testing +3. **Debug with `DEBUG=1`** for detailed output +4. **Promote to unit test** when feature is stable +5. **Add integration tests** for feature combinations + +#### When Fixing Bugs +1. **Create minimal scratch test** reproducing the issue +2. **Debug with `DEBUG=1`** to understand the problem +3. **Fix the issue** and verify with scratch test +4. **Update existing tests** if needed +5. **Clean up scratch tests** after resolution + +### Debugging Tools + +#### Enable Debug Mode +```bash +DEBUG=1 bun run lang.js script.txt ``` -## Usage +This shows: +- Token stream from lexer +- AST structure from parser +- Function call traces +- Scope information -### Running Scripts +#### Call Stack Tracking +The interpreter tracks function calls to detect infinite recursion: ```bash -node lang.js script.txt +# Shows call statistics after execution +bun run lang.js script.txt ``` -### Testing -The project uses a structured testing approach with unit and integration tests: - -#### Unit Tests -Located in `tests/` directory, each focusing on a specific language feature: -- `01_lexer_basic.txt` - Basic lexer functionality -- `02_arithmetic_operations.txt` - Arithmetic operations -- `03_comparison_operators.txt` - Comparison operators -- `04_logical_operators.txt` - Logical operators -- `05_io_operations.txt` - IO operations -- `06_function_definitions.txt` - Function definitions -- `07_case_expressions.txt` - Case expressions and pattern matching -- `08_first_class_functions.txt` - First-class function features -- `09_tables.txt` - Table literals and access -- `10_standard_library.txt` - Standard library functions - -#### Integration Tests -Test combinations of multiple features: -- `integration_01_basic_features.txt` - Basic feature combinations -- `integration_02_pattern_matching.txt` - Pattern matching with other features -- `integration_03_functional_programming.txt` - Functional programming patterns - -#### Running Tests +### Running Tests ```bash # Run all tests ./run_tests.sh -# Run individual tests -node lang.js tests/01_lexer_basic.txt -node lang.js tests/integration_01_basic_features.txt +# Run individual test +bun run lang.js tests/01_lexer_basic.txt + +# Run scratch test +bun run lang.js scratch_tests/test_debug_issue.txt ``` -## Implementation Details +## Architecture Deep Dive -### Architecture -- **Lexer**: Tokenizes input into tokens (numbers, identifiers, operators, etc.) -- **Parser**: Builds Abstract Syntax Tree (AST) from tokens -- **Interpreter**: Executes AST with scope management +### Combinator Foundation +Every operation is implemented as a function call to standard library combinators: -### Key Components -- **Token Types**: Supports all basic operators, literals, and special tokens -- **AST Nodes**: Expression, statement, and declaration nodes -- **Scope Management**: Lexical scoping with proper variable resolution -- **Error Handling**: Comprehensive error reporting for parsing and execution +```javascript +// Arithmetic +x + y → add(x, y) +x - y → subtract(x, y) +x * y → multiply(x, y) -## Recent Fixes +// Comparison +x = y → equals(x, y) +x > y → greaterThan(x, y) -### ✅ Parser Ambiguity with Unary Minus Arguments (Latest Fix) -- **Issue**: `filter @isPositive -3` was incorrectly parsed as binary operation instead of function call with unary minus argument -- **Root Cause**: Parser treating `FunctionReference MINUS` as binary minus operation -- **Solution**: Added special case in `parseExpression()` to handle `FunctionReference MINUS` pattern -- **Status**: ✅ Resolved - Standard library functions now work with negative arguments +// Logical +x and y → logicalAnd(x, y) +not x → logicalNot(x) -### ✅ Unary Minus Operator -- **Issue**: Stack overflow when parsing negative numbers (e.g., `-1`) -- **Root Cause**: Parser lacked specific handling for unary minus operator -- **Solution**: Added `UnaryMinusExpression` parsing and evaluation -- **Status**: ✅ Resolved - All tests passing +// Function application +f x → apply(f, x) +``` -### ✅ IO Operation Parsing -- **Issue**: IO operations not parsed correctly at top level -- **Solution**: Moved IO parsing to proper precedence level -- **Status**: ✅ Resolved +### Standard Library Combinators +The language includes a comprehensive set of combinators: +- **Arithmetic**: `add`, `subtract`, `multiply`, `divide`, `negate` +- **Comparison**: `equals`, `greaterThan`, `lessThan`, etc. +- **Logical**: `logicalAnd`, `logicalOr`, `logicalNot` +- **Higher-Order**: `map`, `compose`, `pipe`, `apply`, `filter` -### ✅ Decimal Number Support -- **Issue**: Decimal numbers not handled correctly -- **Solution**: Updated lexer and interpreter to use `parseFloat()` -- **Status**: ✅ Resolved +### Parser Architecture +The parser uses a **precedence climbing** approach with combinator translation: +1. **Lexer**: Converts source to tokens +2. **Parser**: Builds AST with operator-to-function translation +3. **Interpreter**: Evaluates AST using combinator functions -## Known Issues +## Documentation -### 🔄 Logical Operator Precedence -- **Issue**: Logical operators (`and`, `or`, `xor`) have incorrect precedence relative to function calls -- **Example**: `isEven 10 and isPositive 5` is parsed as `isEven(10 and isPositive(5))` instead of `(isEven 10) and (isPositive 5)` -- **Impact**: Complex expressions with logical operators may not evaluate correctly -- **Status**: 🔄 In Progress - Working on proper operator precedence hierarchy -- **Workaround**: Use parentheses to explicitly group expressions: `(isEven 10) and (isPositive 5)` +### Design Documents +- **[design/README.md](design/README.md)** - Main design documentation entry point +- **[design/PROJECT_ROADMAP.md](design/PROJECT_ROADMAP.md)** - Current status and next steps +- **[design/COMBINATORS.md](design/COMBINATORS.md)** - Combinator foundation explanation +- **[design/ARCHITECTURE.md](design/ARCHITECTURE.md)** - Complete system architecture overview -### 🔄 Parentheses Parsing with Logical Operators -- **Issue**: Some expressions with logical operators inside parentheses fail to parse -- **Example**: `add (multiply 3 4) (isEven 10 and isPositive 5)` may fail with parsing errors -- **Status**: 🔄 In Progress - Related to logical operator precedence issue +### Implementation Guides +- **[design/implementation/IMPLEMENTATION_GUIDE.md](design/implementation/IMPLEMENTATION_GUIDE.md)** - Current implementation guide +- **[design/implementation/COMPLETED_FEATURES.md](design/implementation/COMPLETED_FEATURES.md)** - Summary of completed features -## Development +### Historical Documentation +- **[design/HISTORY/](design/HISTORY/)** - Resolved issues and completed work -### File Structure -``` -. -├── lang.js # Main implementation -├── test.txt # Comprehensive test file -├── tests/ # Unit and integration tests -│ ├── 01_lexer_basic.txt -│ ├── 02_arithmetic_operations.txt -│ ├── ... -│ ├── integration_01_basic_features.txt -│ ├── integration_02_pattern_matching.txt -│ └── integration_03_functional_programming.txt -├── run_tests.sh # Test runner script -├── FIXME.md # Issues and fixes documentation -└── README.md # This file -``` +## Contributing -### Debugging -Enable debug mode by setting `DEBUG=true`: +### Development Guidelines +1. **Follow the combinator approach** for new operations +2. **Use scratch tests** for rapid prototyping +3. **Promote to unit tests** when features are stable +4. **Maintain backward compatibility** - existing code must work +5. **Document changes** in design documents + +### Code Style +- **Functional approach**: Prefer pure functions +- **Combinator translation**: All operations become function calls +- **Clear naming**: Descriptive function and variable names +- **Comprehensive testing**: Test edge cases and combinations + +### Testing Requirements +- **Unit tests** for all new features +- **Integration tests** for feature combinations +- **Backward compatibility** tests for existing code +- **Edge case coverage** for robust implementation + +## Quick Reference + +### Key Files +- **lang.js**: Main interpreter with standard library +- **parser.js**: Parser with combinator translation +- **lexer.js**: Tokenizer with all operators +- **tests/**: Comprehensive test suite + +### Development Commands ```bash -DEBUG=true node lang.js script.txt +# Run all tests +./run_tests.sh + +# Run with debug +DEBUG=1 node lang.js script.txt + +# Run individual test +node lang.js tests/01_lexer_basic.txt ``` -## Contributing +--- -1. Create focused unit tests for new features -2. Add integration tests for feature combinations -3. Update documentation -4. Run the full test suite before submitting changes \ No newline at end of file +This language demonstrates how **functional programming principles** can solve real parsing problems while maintaining intuitive syntax. The combinator foundation provides a solid base for building powerful abstractions. \ No newline at end of file |