diff options
Diffstat (limited to 'js/scripting-lang/baba-yaga-c/ROADMAP.md')
| -rw-r--r-- | js/scripting-lang/baba-yaga-c/ROADMAP.md | 891 |
1 files changed, 143 insertions, 748 deletions
diff --git a/js/scripting-lang/baba-yaga-c/ROADMAP.md b/js/scripting-lang/baba-yaga-c/ROADMAP.md index 2d8b827..963c46f 100644 --- a/js/scripting-lang/baba-yaga-c/ROADMAP.md +++ b/js/scripting-lang/baba-yaga-c/ROADMAP.md @@ -1,748 +1,143 @@ -# Baba Yaga C Implementation - Complete Roadmap - -## Executive Summary - -**Current Status**: ✅ **Major Breakthrough Achieved** - Infix operator system now working -**Overall Progress**: ~85% Complete -**Test Results**: 26/26 basic tests PASS, 51/66 standard library tests PASS, 5/27 JavaScript tests PASS -**Critical Fix Applied**: ✅ **Parser precedence system now active** - infix operators working -**Estimated Completion**: 2-3 days (function call execution and pattern matching fixes) - -## Language Overview - -Baba Yaga is a functional programming language with combinator-based architecture where all operators translate to function calls. The C implementation provides a high-performance, memory-efficient interpreter that maintains compatibility with the JavaScript reference implementation. - -### Key Features -- **Function Application**: Juxtaposition-based (`f x` means `apply(f, x)`) -- **Function References**: `@` operator for function references and expressions (`@function` and `@(expression)`) -- **Pattern Matching**: `when` expressions for conditional logic with wildcard support -- **User-Defined Functions**: `name : params -> body;` syntax -- **Tables**: APL-inspired data structures (planned) -- **Combinator Foundation**: All operations become function calls -- **Multiple Statements**: Semicolon-separated statement sequences - - - -## Current Implementation Status - -### ✅ COMPLETED - Core Functionality - -#### Phase 1: Foundation (COMPLETE) -- **Value System**: ✅ Complete with all data types (number, string, boolean, table, function, nil) -- **Table Implementation**: ✅ Immutable hash table with reference counting -- **Memory Management**: ✅ Reference counting for tables and functions -- **Function Representation**: ✅ Native and user function support - -#### Phase 2: Lexer (COMPLETE) -- **Tokenization**: ✅ Complete with all token types including IO operations -- **Special Cases**: ✅ Unary/binary minus handling, function references (`@function`) -- **Error Handling**: ✅ Line/column information in error messages -- **Performance**: ✅ Efficient tokenization for large files - -#### Minus/Unary Minus/Infix Minus Token Distinction -- **Background**: The JavaScript reference implementation and the C version both encountered difficulty distinguishing between infix minus (binary subtraction), unary minus (negation), and ambiguous minus tokens during parsing. This is a classic challenge in language design, especially for functional/juxtaposition-based languages. -- **Tokenization**: The lexer is designed to emit `TOKEN_OP_MINUS` for minus, and could be extended to emit `TOKEN_OP_UNARY_MINUS` or `TOKEN_OP_INFIX_MINUS` if context is available. Currently, the C lexer emits only `TOKEN_OP_MINUS` and leaves the distinction to the parser (see TODO in lexer.c). -- **Parser Handling**: The parser is responsible for distinguishing between unary and binary minus based on context and operator precedence. This is handled in the precedence chain and by lookahead logic. -- **Current Status**: This minus distinction is not the cause of current parsing or test failures (e.g., with `when` expressions or multi-statement files). The parser and lexer are set up to handle this distinction, and the main parser/lexer bug was unrelated to minus handling. -- **Future Work**: If failures are observed specifically with negative numbers or ambiguous minus usage, revisit this logic. For now, the current approach is sufficient, but future contributors should be aware of this subtlety. - -#### Phase 3: Parser (COMPLETE) -- **AST Construction**: ✅ Complete AST node types for all language constructs -- **Operator Precedence**: ✅ Proper precedence handling with combinator translation -- **Combinator Translation**: ✅ All operators translate to function calls -- **Pattern Matching**: ✅ Complete pattern parsing with `when` expressions -- **Function Definitions**: ✅ User-defined function parsing with `->` syntax -- **Function References**: ✅ `@` operator parsing for both `@function` and `@(expression)` - -#### Phase 4: Interpreter Foundation (COMPLETE) -- **Scope Management**: ✅ Complete scope system with variable lookup and assignment -- **Function Calling**: ✅ Native function calling with proper argument passing -- **Standard Library**: ✅ Complete standard library with 20+ functions -- **Error Handling**: ✅ Runtime error detection and reporting infrastructure - -#### Phase 5: Enhanced Interpreter (COMPLETE) -- **Full AST Execution**: ✅ Complete expression evaluation pipeline -- **Function References**: ✅ @ operator support for function references and expressions -- **Pattern Matching**: ✅ Complete `when` expression evaluation with wildcard support -- **User-Defined Functions**: ✅ Function creation, parameter binding, and execution -- **Basic Integration**: ✅ Complete lexer → parser → interpreter pipeline - -#### Phase 6: Multiple Statement Support (COMPLETE) -- **Sequence Parsing**: ✅ Parse semicolon-separated statements into sequence nodes -- **Statement Execution**: ✅ Execute statements in sequence, return last value -- **Variable Scoping**: ✅ Variables defined in earlier statements available in later ones -- **Real-world Usage**: ✅ Language now supports practical multi-statement programs - -## 🔴 CRITICAL ISSUES TO FIX - -### 1. Multiple Statement Parsing (CRITICAL) ✅ **COMPLETED** -**Current Status**: ✅ All statements executed in sequence -**Impact**: ✅ Unblocks real-world usage -**Priority**: ✅ RESOLVED -**Effort**: ✅ Completed - -**Implementation**: -- ✅ Implemented `NODE_SEQUENCE` AST node type -- ✅ Created `parser_parse_statements()` for semicolon-separated parsing -- ✅ Updated interpreter to execute statement sequences -- ✅ Added sequence node accessor functions -- ✅ Updated test suite to reflect working functionality - -**Example Working**: -```c -// ✅ "x : 10; y : 20; add x y" → executes all three statements, returns 30 -// ✅ "a : 5; b : 3; c : 2; add a @multiply b c" → returns 11 -``` - -### 2. JavaScript Test Suite Compatibility (HIGH PRIORITY) 🔄 **MAJOR PROGRESS** -**Current Status**: 5/27 tests pass (19% success rate) - **Infrastructure now working** -**Priority**: HIGH -**Effort**: 1-2 days - -**Major Achievements**: - -#### Infix Operators (CRITICAL - 15+ tests) ✅ **FIXED - PARSER PRECEDENCE SYSTEM NOW ACTIVE** -- **Arithmetic**: `a + b`, `a - b`, `a * b`, `a / b`, `a % b`, `a ^ b` ✅ **WORKING** -- **Logical**: `a and b`, `a or b`, `a xor b`, `not a` ✅ **WORKING** -- **Comparison**: `a = b`, `a > b`, `a < b`, `a >= b`, `a <= b`, `a != b` ✅ **WORKING** -- **Status**: ✅ **PARSER PRECEDENCE SYSTEM NOW ACTIVE** -- **Fix Applied**: Changed main `parser_parse_expression()` to call `parser_parse_logical()` instead of `parser_parse_application()` -- **Additional Fix**: Modified lexer to treat `and`, `or`, `xor` as identifiers instead of keywords for dual function/operator support -- **Impact**: Unblocks tests 01, 02, 03, 04, 05, 08, 11, 13, 14, 15, 16, 18, 19, 20, 22, 23 -- **Effort**: ✅ **COMPLETED** (2 hours) -- **Precedence Chain**: `logical` → `comparison` → `additive` → `multiplicative` → `power` → `primary` ✅ **ACTIVE** -- **All Operator Tokens**: ✅ Already implemented in lexer -- **All Operator Functions**: ✅ Already implemented in parser -- **Interpreter Support**: ✅ Already handles binary operations - -#### Pattern Matching (Critical - 8+ tests) ✅ **COMPLETED** -- `when` expressions: `when x is 42 then "correct" _ then "wrong"` -- **Status**: ✅ Fully implemented and working -- **Features**: Basic patterns, multiple patterns, wildcard (`_`) support, variable assignment context -- **Fix Applied**: Added `TOKEN_KEYWORD_WHEN`, `TOKEN_KEYWORD_IS`, `TOKEN_KEYWORD_THEN` to function call parsing stop tokens -- **Impact**: Unblocks tests 01, 07, 21, and integration tests -- **Result**: Pattern matching now works in all contexts - -#### Multi-Pattern `when` Expressions and Wildcard Support (IN PROGRESS) -- **Goal:** Support multi-pattern `when` expressions, e.g.: - ``` - result : when x is 42 then "correct" _ then "wrong"; - ``` -- **Reference:** The JavaScript implementation and its AST output, which treats each `pattern then result` as a separate case, with `_` as a wildcard pattern. -- **C Implementation:** The parser is structured to call `parser_parse_when_pattern` for each pattern/result pair in a loop within `parser_parse_when_expression`. -- **Progress:** - - Lexer correctly tokenizes `_` as a `TOKEN_IDENTIFIER` with lexeme "_". - - `parser_parse_when_pattern` recognizes `_` as a wildcard and creates a special literal node. - - Result parsing now uses `parser_parse_when_result_expression` to avoid greedy token consumption. - - Debug output confirms the parser is correctly positioned at the `_` token for the second pattern. -- **Current Issue:** - - After parsing the first pattern/result, the parser is not calling `parser_parse_when_pattern` for the second pattern. - - Instead, the `_` token is being consumed by a lower-level expression parser, not the wildcard logic. - - The loop in `parser_parse_when_expression` is likely breaking too early or not recognizing the next pattern start correctly. -- **Next Steps:** - 1. Add debug output at the end of the loop in `parser_parse_when_expression` to confirm loop behavior. - 2. Fix the loop condition if necessary so that it always calls `parser_parse_when_pattern` for each pattern/result pair. - 3. Once fixed, verify with tests and document the solution. - -#### Function Definitions (High - 3+ tests) ✅ **COMPLETED** -- User-defined functions: `name : params -> body;` -- **Status**: ✅ Fully implemented and working -- **Features**: Function definition parsing, function storage, parameter binding, function execution -- **Working**: Identity functions, simple parameter binding, `@` operator support, complex function bodies -- **Fix Applied**: Modified `baba_yaga_function_call()` to accept scope parameter and pass current scope as parent -- **Impact**: Unblocks basic function tests (test 06 passes) and complex function bodies now work -- **Result**: Function definitions work for all cases including complex function bodies - -#### Tables (High - 5+ tests) -- Data structures: `data : {a: 1, b: 2};` -- **Impact**: Blocks tests 09, 12, 17 -- **Effort**: 3-4 hours - -#### Advanced Features (Medium - 8+ tests) -- Each combinator, embedded functions, via operator, etc. -- **Impact**: Blocks tests 18-23 -- **Effort**: 3-4 hours - -**Tasks**: -- [x] Implement pattern matching (`when` expressions) ✅ -- [x] Debug variable assignment context for `when` expressions ✅ -- [x] Add user-defined function syntax (`->`) ✅ -- [x] Implement `@` operator for function references and expressions ✅ -- [x] Debug function calls within function bodies (CRITICAL - blocks complex functions) ✅ -- [x] Implement infix operators (CRITICAL - blocks 15+ tests) ✅ **COMPLETED** -- [ ] Fix function call execution for logical operators (HIGH - blocks 12 standard library tests) -- [ ] Fix pattern matching parsing issues (HIGH - blocks multiple JS tests) -- [ ] Implement table data structures -- [ ] Add advanced functional programming features (each combinator, via operator, etc.) - -### 3. Standard Library Issues (MEDIUM PRIORITY) 🔄 **MAJOR PROGRESS** -**Current Status**: 51/66 tests pass (77% success rate) - **Improved from previous** -**Priority**: MEDIUM -**Effort**: 1-2 days - -**Major Achievements**: - -#### Debug Output Control ✅ **COMPLETED** -- **Problem**: Debug output was hardcoded `printf` statements, not using debug system -- **Fix Applied**: Replaced all hardcoded debug output with proper `DEBUG_TRACE()` macros -- **Result**: Debug output now properly controlled by `DEBUG` environment variable -- **Impact**: Test suites now run cleanly without debug output interference - -#### Logical Operator Parsing ✅ **COMPLETED** -- **Problem**: Logical operators (`and`, `or`, `xor`) were treated as keywords only, not function names -- **Fix Applied**: Modified lexer to treat `and`, `or`, `xor` as identifiers instead of keywords -- **Result**: Logical operators now work both as infix operators AND function calls -- **Impact**: Logical operator tests now parse correctly (though execution needs fixing) - -**Current Failures**: - -#### Function Call Execution (12 failures) 🔴 **HIGH PRIORITY** -- **Problem**: Logical operator function calls return `<function:and>` instead of executing -- **Example**: `and true true` returns `<function:and>` instead of `true` -- **Impact**: Blocks 12 standard library tests -- **Root Cause**: Function call parsing works, but execution not happening properly - -#### Type Checking (1 failure) ⚠️ **MEDIUM PRIORITY** -- `less` function: Returns `<function:less>` instead of error message -- **Impact**: 1 standard library test fails - -#### Precision Issues (2 failures) ⚠️ **LOW PRIORITY** -- `pow` function: Square root precision (1 digit difference) -- Negative power: Still fails due to negative number parsing issue -- **Impact**: 2 standard library tests fail - -**Remaining Tasks**: -- [ ] Fix function call execution for logical operators (HIGH - blocks 12 tests) -- [ ] Fix type checking for `less` function (MEDIUM - blocks 1 test) -- [ ] Fix negative number parsing for `pow 2 -1` case (LOW) -- [ ] Minor precision adjustment for square root (LOW - 1 digit difference) - -## 🟡 MEDIUM PRIORITY ISSUES - -### 3. User-Defined Functions (MEDIUM) -**Current Status**: Not implemented -**Priority**: MEDIUM -**Effort**: 3-4 days - -**Required Features**: -- Function body storage in AST -- Parameter binding during calls -- Local scope creation -- Recursive function support - -**Tasks**: -- [ ] Implement function body storage in AST -- [ ] Add parameter binding during function calls -- [ ] Create local scope creation for function execution -- [ ] Support recursive function calls -- [ ] Add stack overflow protection - -### 4. Pattern Matching Evaluation (MEDIUM) -**Current Status**: Basic parsing complete, evaluation pending -**Priority**: MEDIUM -**Effort**: 2-3 days - -**Required Features**: -- Boolean expression evaluation in patterns -- Multiple value patterns -- Pattern guards - -**Tasks**: -- [ ] Implement pattern matching engine -- [ ] Add boolean expression evaluation in patterns -- [ ] Support multiple value patterns -- [ ] Add wildcard pattern support -- [ ] Implement enhanced case statements - -### 5. Enhanced Error Handling (MEDIUM) -**Current Status**: Basic error detection -**Priority**: MEDIUM -**Effort**: 1-2 days - -**Required Features**: -- Specific error messages with context -- Line/column numbers in errors -- Source code snippets -- Stack traces for function calls - -**Tasks**: -- [ ] Add specific error messages (division by zero, undefined variable, etc.) -- [ ] Add error context (line/column numbers, source snippets) -- [ ] Add error recovery where possible -- [ ] Improve error message formatting - -## 🔵 LOW PRIORITY ISSUES - -### 6. Memory Leak Testing (LOW) -**Current Status**: Not tested -**Priority**: LOW -**Effort**: 1 day - -**Tasks**: -- [ ] Add valgrind to build system -- [ ] Create `make memcheck` target -- [ ] Test all components for leaks -- [ ] Fix any memory issues found -- [ ] Add memory usage monitoring - -### 7. Performance Optimization (LOW) -**Current Status**: Basic performance -**Priority**: LOW -**Effort**: 1-2 days - -**Tasks**: -- [ ] Profile critical execution paths -- [ ] Optimize memory allocation patterns -- [ ] Add performance benchmarks -- [ ] Optimize AST accessor functions - -### 8. Documentation and Testing (LOW) -**Current Status**: Basic documentation -**Priority**: LOW -**Effort**: 1 day - -**Tasks**: -- [ ] Complete API documentation -- [ ] Add comprehensive test suite -- [ ] Create architecture documentation -- [ ] Add usage examples - -## Current State - For New Team Members - -### What's Working ✅ - -#### Core Language Features -- **Basic Expressions**: Numbers, strings, booleans, variables -- **Arithmetic Operations**: `add`, `subtract`, `multiply`, `divide`, `modulo`, `pow` -- **Comparison Operations**: `equals`, `not_equals`, `less`, `greater`, etc. -- **Logical Operations**: `and`, `or`, `xor`, `not` -- **Function Calls**: Native function calling with proper argument evaluation -- **Function References**: `@function` and `@(expression)` syntax -- **Variable Assignment**: `name : value;` syntax -- **Multiple Statements**: Semicolon-separated sequences execute correctly - -#### Advanced Features -- **Pattern Matching**: `when` expressions with wildcard (`_`) support -- **User-Defined Functions**: `name : params -> body;` syntax -- **Higher-Order Functions**: `apply`, `compose` working correctly -- **IO Operations**: `..out`, `..assert` working correctly - -#### Test Results -- **Basic Tests**: 26/26 PASS (100%) -- **Standard Library**: 60/66 PASS (91%) -- **JavaScript Compatibility**: 4/27 PASS (15%) - -### What's Partially Working ⚠️ - -#### User-Defined Functions -- ✅ Function definition parsing works -- ✅ Function storage in scope works -- ✅ Parameter binding works -- ✅ Simple function bodies work (e.g., `x -> x`) -- ❌ Complex function bodies fail (e.g., `x y -> add x y` returns `nil`) - -**Example Working**: -```c -identity_func : x -> x; // ✅ Works -identity_func 42; // ✅ Returns 42 -``` - -**Example Failing**: -```c -add_func : x y -> add x y; // ❌ Function body returns nil -add_func 3 4; // ❌ Returns nil instead of 7 -``` - -### What's Not Implemented ❌ - -#### Tables -- Table literals: `{a: 1, b: 2}` -- Table access: `table.key` or `table["key"]` -- Table operations: `keys`, `values`, `size` - -#### Advanced Functional Features -- Each combinator: `each function list` -- Via operator: `via function value` -- Embedded functions: Functions within expressions -- Function composition with multiple functions - -#### Syntax Differences from JavaScript -- **Assertion Syntax**: JavaScript uses `..assert sum = 13`, C uses `..assert equals sum 13` - -### What's Partially Implemented 🔧 - -#### Function Call Execution (HIGH PRIORITY) -- **Status**: 🔧 **FUNCTION CALL PARSING WORKS BUT EXECUTION FAILS** -- **Problem**: Logical operator function calls return `<function:and>` instead of executing -- **Impact**: Blocks 12 standard library tests -- **Example**: `and true true` returns `<function:and>` instead of `true` -- **Root Cause**: Function call parsing works, but execution not happening properly - -#### Pattern Matching Parsing (HIGH PRIORITY) -- **Status**: 🔧 **PATTERN MATCHING EVALUATION WORKS BUT PARSING HAS ISSUES** -- **Problem**: "Expected 'then' after pattern in when case" errors -- **Impact**: Blocks multiple JavaScript compatibility tests -- **Root Cause**: Parser logic for `when` expressions needs refinement - -### Critical Issues to Address - -#### 1. Function Call Execution (HIGH PRIORITY) 🔴 **BLOCKING** -**Problem**: Logical operator function calls return `<function:and>` instead of executing -**Root Cause**: Function call parsing works, but execution not happening properly -**Impact**: Blocks 12 standard library tests -**Fix**: Investigate function call execution in interpreter -**Effort**: 2-3 hours -**Files to Check**: `src/interpreter.c` function call evaluation - -#### 2. Pattern Matching Parsing (HIGH PRIORITY) 🔴 **BLOCKING** -**Problem**: "Expected 'then' after pattern in when case" errors -**Root Cause**: Parser logic for `when` expressions needs refinement -**Impact**: Blocks multiple JavaScript compatibility tests -**Fix**: Debug and fix `when` expression parsing logic -**Effort**: 2-3 hours -**Files to Check**: `src/parser.c` when expression parsing - -#### 3. Logical Operator Parsing (MEDIUM PRIORITY) ✅ **COMPLETED** -**Problem**: `and`, `or`, `xor` keywords parsed as function calls instead of operators -**Root Cause**: Logical operators were treated as keywords only, not function names -**Impact**: ✅ **RESOLVED** - Now works as both infix operators and function calls -**Fix**: ✅ Modified lexer to treat `and`, `or`, `xor` as identifiers instead of keywords -**Effort**: ✅ **COMPLETED** (1 hour) -**Files Modified**: `src/lexer.c` keyword handling, `src/parser.c` logical operator handling - -#### 4. Function Calls in Function Bodies (MEDIUM PRIORITY) ✅ **COMPLETED** -**Problem**: Function calls like `add x y` within function bodies return `nil` -**Root Cause**: Function call arguments are not being evaluated to their values -**Impact**: ✅ **RESOLVED** - Complex user-defined functions now work -**Files Modified**: `src/interpreter.c` function call evaluation, `src/function.c` user function execution - -#### 5. Table Implementation (MEDIUM PRIORITY) -**Problem**: Table data structures not implemented -**Impact**: Blocks tests 09, 12, 17 -**Files to Check**: `src/table.c` (exists but needs implementation) - -#### 6. Advanced Combinators (MEDIUM PRIORITY) -**Problem**: Each combinator, via operator not implemented -**Impact**: Blocks tests 18-23 -**Files to Check**: `src/stdlib.c` for combinator implementations - -### Code Architecture - -#### Key Files -- `src/lexer.c`: Tokenization (✅ Complete) -- `src/parser.c`: AST construction (✅ Complete) -- `src/interpreter.c`: Expression evaluation (✅ Complete) -- `src/function.c`: Function management (✅ Complete) -- `src/scope.c`: Variable scoping (✅ Complete) -- `src/stdlib.c`: Standard library (✅ Complete) -- `src/table.c`: Table implementation (❌ Needs work) - -#### Key Data Structures -- `Value`: Union type for all values (number, string, boolean, function, table, nil) -- `ASTNode`: Abstract syntax tree nodes -- `Scope`: Variable scope with parent-child relationships -- `FunctionValue`: Function representation with native/user distinction - -#### Key Functions -- `baba_yaga_execute()`: Main execution entry point -- `interpreter_evaluate_expression()`: Core evaluation logic -- `baba_yaga_function_call()`: Function calling mechanism -- `scope_define()` / `scope_get()`: Variable management - -### Testing Strategy - -#### Test Suites -- `run_tests.sh`: Basic functionality tests (26 tests, all pass) -- `test_stdlib.sh`: Standard library tests (66 tests, 60 pass) -- `run_comprehensive_tests.sh`: JavaScript compatibility tests (27 tests, 4 pass) - -#### Debugging Commands -```bash -# Build with debug info -make debug - -# Test specific features -./bin/baba-yaga 'add 3 4;' # Basic function call -./bin/baba-yaga '@(add 3 4);' # Function reference -./bin/baba-yaga 'x : 42; x;' # Variable assignment -./bin/baba-yaga 'when 42 is 42 then "yes";' # Pattern matching -./bin/baba-yaga 'f : x -> x; f 42;' # User-defined function - -# Run test suites -./run_tests.sh -./test_stdlib.sh -./run_comprehensive_tests.sh -``` - -## Implementation Roadmap - -### Week 1: Critical Fixes ✅ **MAJOR PROGRESS** - -#### Days 1-2: Infix Operator Implementation (CRITICAL) ✅ **COMPLETED** -**Dependencies**: None -**Success Criteria**: -- `"a + b"`, `"a and b"`, `"a = b"` parse and execute correctly ✅ **ACHIEVED** -- Operator precedence and associativity work properly ✅ **ACHIEVED** -- `@(a + b)` syntax works with infix expressions ✅ **ACHIEVED** -- 15+ JavaScript compatibility tests pass ✅ **INFRASTRUCTURE READY** - -#### Days 3-5: Function Call Execution and Pattern Matching (HIGH) 🔄 **IN PROGRESS** -**Dependencies**: Infix operators ✅ **COMPLETED** -**Success Criteria**: -- All 66 standard library tests pass (currently 51/66) -- Function call execution works for logical operators -- Pattern matching parsing issues resolved -- Higher-order functions work correctly ✅ **ACHIEVED** -- IO operations behave as expected ✅ **ACHIEVED** -- Type errors are properly reported ✅ **ACHIEVED** - -### Week 2: Advanced Features - -#### Days 1-3: Table Implementation and Advanced Features (MEDIUM) -**Dependencies**: Function call execution and pattern matching fixes -**Success Criteria**: -- Basic table operations work (`{a: 1, b: 2}`) -- Table access and manipulation functions work -- Advanced combinators (each, via) implemented -- User-defined functions work correctly ✅ **ACHIEVED** -- Parameters are properly bound ✅ **ACHIEVED** -- Local variables work correctly ✅ **ACHIEVED** - -#### Days 4-5: Polish and Optimization (MEDIUM/LOW) -**Dependencies**: Table implementation and advanced features -**Success Criteria**: -- Pattern matching parsing issues resolved ✅ **INFRASTRUCTURE READY** -- Boolean patterns evaluate correctly ✅ **ACHIEVED** -- Multi-parameter patterns work ✅ **ACHIEVED** -- Pattern guards function properly ✅ **ACHIEVED** -- No memory leaks detected -- Comprehensive error messages ✅ **ACHIEVED** - -## Test Results Analysis - -### ✅ Passing Tests (82/119 = 69%) -- **Basic Functionality**: 26/26 tests PASS (100%) -- **Standard Library**: 51/66 tests PASS (77%) -- **JavaScript Compatibility**: 5/27 tests PASS (19%) - -#### Detailed Breakdown - -**Basic Tests (26/26 PASS)**: -- Arithmetic operations, function calls, variable assignment -- Multiple statements, higher-order functions, IO operations -- Error handling, pattern matching basics - -**Standard Library Tests (51/66 PASS)**: -- Core arithmetic: `add`, `subtract`, `multiply`, `divide`, `modulo`, `pow` ✅ -- Comparison: `equals`, `not_equals`, `less`, `greater`, etc. ✅ -- Logical: `and`, `or`, `xor`, `not` 🔧 (parsing works, execution needs fix) -- Higher-order: `apply`, `compose` ✅ -- IO: `..out`, `..assert` ✅ - -**JavaScript Compatibility Tests (5/27 PASS)**: -- ✅ `02_arithmetic_operations`: Basic arithmetic working -- ✅ `04_logical_operators`: Logical operations working -- ✅ `06_function_definitions`: Basic function definitions working -- ✅ `10_standard_library`: Standard library working -- ✅ `20_via_operator`: Via operator working - -### ❌ Failing Tests (37/119 = 31%) - -#### Standard Library Failures (15/66): -- **Function Call Execution**: 12 failures due to logical operators returning functions instead of executing -- **Precision**: 2 failures (square root 1-digit difference, negative power parsing) -- **Type Checking**: 1 failure (less function type checking) - -#### JavaScript Compatibility Failures (22/27): -- **Pattern Matching**: 8+ failures due to "Expected 'then' after pattern in when case" errors -- **Some Infix Operators**: 7+ failures due to remaining parsing issues -- **Missing Features**: Tables, advanced combinators, embedded functions -- **Function Body Issues**: ✅ Fixed - complex function bodies now work -- **Infix Operator Infrastructure**: ✅ Fixed - parser precedence system now active - -## Success Metrics - -### Current Status -- ✅ Basic functionality: 26/26 tests PASS (100%) -- ✅ Standard library: 51/66 tests PASS (77%) -- ✅ Multiple statements: Fully working -- ✅ Higher-order functions: Fully working -- ✅ IO operations: Fully working -- ✅ Pattern matching evaluation: Fully working -- ✅ Function definitions: All cases working -- ✅ Function references: `@` operator working -- ✅ Error handling: Type checking implemented -- ✅ Infix operators: **PARSER PRECEDENCE SYSTEM NOW ACTIVE** ✅ **MAJOR BREAKTHROUGH** -- 🔧 Function call execution: Logical operators return functions instead of executing (blocks 12 tests) -- 🔧 Pattern matching parsing: "Expected 'then' after pattern" errors (blocks multiple JS tests) -- ❌ Tables: Not implemented -- ❌ Advanced combinators: Not implemented - -### Target Status (End of Week 1) -- ✅ All basic functionality: 26/26 tests PASS (100%) -- ✅ Standard library: 66/66 tests PASS (100%) -- ✅ Function definitions: All cases working ✅ **ACHIEVED** -- ✅ Infix operators: Full implementation (arithmetic, logical, comparison) ✅ **ACHIEVED** -- 🔧 Function call execution: Logical operators execute properly (currently return functions) -- 🔧 Pattern matching parsing: All `when` expressions parse correctly -- ✅ Tables: Basic implementation -- ✅ Advanced combinators: Core implementation -- ✅ JavaScript compatibility: 20+/27 tests PASS -- ✅ Error handling: Comprehensive ✅ **ACHIEVED** -- ✅ Memory management: Leak-free - -## Quick Start Commands - -### Build and Test -```bash -# Build with debug info -make debug - -# Test basic functionality (all working ✅) -./bin/baba-yaga '5 + 3;' # Should output: 8 -./bin/baba-yaga '10 - 3;' # Should output: 7 -./bin/baba-yaga '6 * 7;' # Should output: 42 -./bin/baba-yaga 'x : 42; x;' # Should output: 42 -./bin/baba-yaga 'add 5 3;' # Should output: 8 -./bin/baba-yaga '@multiply 2 3;' # Should output: 6 -./bin/baba-yaga 'add 5 @multiply 3 4;' # Should output: 17 - -# Multiple statements now working ✅ -./bin/baba-yaga 'x : 10; y : 20; add x y;' # ✅ Executes all statements, returns 30 - -# Run test suite -./run_tests.sh -./test_stdlib.sh - -# Check for memory leaks -valgrind --leak-check=full ./bin/baba-yaga '5 + 3;' -``` - -## Risk Assessment - -### High Risk -- **Memory Leaks**: Complex memory management in C -- **Mitigation**: Comprehensive testing with valgrind - -### Medium Risk -- **Performance Issues**: Can optimize after functionality complete -- **Cross-Platform Issues**: Test on multiple platforms - -### Low Risk -- **Complex Nested Function References**: Known limitation, not blocking -- **Multiple Statement Parsing**: ✅ Completed successfully - -## ✅ MAJOR BREAKTHROUGH: Parser Precedence System Now Active - -### The Achievement -We successfully **activated the infix operator parsing system** that was already fully implemented but not being used. The parser now uses the complete operator precedence chain: - -``` -parser_parse_logical() → parser_parse_comparison() → parser_parse_additive() → -parser_parse_multiplicative() → parser_parse_power() → parser_parse_primary() -``` - -### The Fix Applied -Changed the main `parser_parse_expression()` function to call `parser_parse_logical()` instead of `parser_parse_application()`: - -```c -// Changed from: -return parser_parse_application(parser); -// To: -return parser_parse_logical(parser); -``` - -### Additional Fix: Dual Function/Operator Support -Modified the lexer to treat `and`, `or`, `xor` as identifiers instead of keywords, allowing them to work both as: -- **Infix operators**: `true and false` ✅ **WORKING** -- **Function calls**: `and true true` 🔧 **PARSING WORKS, EXECUTION NEEDS FIX** - -### Impact Achieved -This change immediately: -- ✅ Enabled all infix operators: `a + b`, `a and b`, `a = b`, etc. -- ✅ Support operator precedence: `a + b * c` → `a + (b * c)` -- ✅ Support function references with infix: `@(a + b)` -- ✅ Unblocked infrastructure for 15+ JavaScript compatibility tests - -## Conclusion - -The Baba Yaga C implementation has achieved a **major breakthrough** with the activation of the infix operator system. We now have 69% of functionality working correctly, with the critical infrastructure in place for rapid progress. Major language features including pattern matching, function definitions, function references, and now infix operators are fully implemented and working. - -### Key Achievements -- ✅ **Pattern Matching**: Complete `when` expression support with wildcard patterns -- ✅ **Function Definitions**: User-defined functions with parameter binding -- ✅ **Function References**: `@` operator for both `@function` and `@(expression)` syntax -- ✅ **Multiple Statements**: Semicolon-separated statement sequences -- ✅ **Infix Operators**: **MAJOR BREAKTHROUGH** - Parser precedence system now active -- ✅ **Core Language**: All basic operations and standard library functions working -- ✅ **Debug System**: Proper debug output control implemented - -### Remaining Work -- 🔧 **Function Call Execution**: Fix logical operator function calls returning functions instead of executing -- 🔧 **Pattern Matching Parsing**: Fix "Expected 'then' after pattern" parsing errors -- 📊 **Tables**: Implement table data structures and operations -- 🔄 **Advanced Combinators**: Implement each combinator, via operator, etc. -- 🎯 **JavaScript Compatibility**: Address remaining syntax differences and missing features - -The implementation follows the same architecture as the JavaScript version, ensuring consistency and maintainability. The C version provides better performance and memory efficiency while maintaining the same language semantics. - -**Next Action**: Fix function call execution for logical operators to unblock 12 standard library tests. - -**Estimated completion for full implementation**: 2-3 days remaining. - -## Immediate Next Steps for New Team Members - -### 1. Fix Function Call Execution (HIGH PRIORITY - 2-3 hours) 🔴 **BLOCKING** -**Problem**: Logical operator function calls return `<function:and>` instead of executing - -**Root Cause**: Function call parsing works, but execution not happening properly - -**Implementation Steps**: -1. **Investigate function call execution** in `src/interpreter.c` -2. **Debug why logical operators return functions** instead of executing them -3. **Test function calls**: `and true true`, `or false true`, `xor true false` -4. **Verify execution**: Should return boolean results, not function references - -**Files to Check**: -- `src/interpreter.c`: Function call evaluation logic -- `src/function.c`: Function execution implementation - -**Expected Result**: 12 standard library tests should immediately pass - -### 2. Fix Pattern Matching Parsing (HIGH PRIORITY - 2-3 hours) 🔴 **BLOCKING** -**Problem**: "Expected 'then' after pattern in when case" errors - -**Root Cause**: Parser logic for `when` expressions needs refinement - -**Implementation Steps**: -1. **Debug `when` expression parsing** in `src/parser.c` -2. **Fix pattern boundary detection** logic -3. **Test pattern matching**: `when x is 42 then "yes" _ then "no"` -4. **Verify parsing**: Should parse without "Expected 'then'" errors - -**Files to Check**: -- `src/parser.c`: When expression parsing logic - -**Expected Result**: Multiple JavaScript compatibility tests should pass - -### 2. Implement Tables (MEDIUM - 2-3 days) -**Problem**: Table literals and operations not implemented - -**Implementation Steps**: -1. Implement table literal parsing in `src/parser.c` -2. Add table operations to `src/table.c` -3. Update `src/stdlib.c` with table functions (`keys`, `values`, `size`) -4. Test with: `data : {a: 1, b: 2}; data.a` - -**Files to Modify**: -- `src/parser.c`: Add `NODE_TABLE` parsing -- `src/table.c`: Implement table operations -- `src/stdlib.c`: Add table utility functions - -### 3. Implement Advanced Combinators (MEDIUM - 2-3 days) -**Problem**: Each combinator, via operator not implemented - -**Implementation Steps**: -1. Add `each` combinator to `src/stdlib.c` -2. Add `via` operator support -3. Test with: `each add [1, 2, 3]` \ No newline at end of file +# Baba Yaga C Implementation Roadmap + +## Current Status +- ✅ **Core Language**: Complete and stable (25/27 tests passing) +- ✅ **Table Pattern Matching**: Fixed and working +- ✅ **When Expressions**: Fixed and working +- ✅ **Computed Table Keys**: Fixed and working (Task 1.1 complete) +- ✅ **Multi-value Pattern Expressions**: Fixed and working (Task 1.2 complete) +- ✅ **Pattern Matching Memory**: Fixed and working (Task 1.3 complete) +- ✅ **Partial Application Support**: Fixed and working (Task 2.3 complete) +- ❌ **2 Remaining Issues**: Test 22 parser issue, Integration Test 02 file reading issue + +## Quick Reference +- **Test Command**: `./bin/baba-yaga tests/22_parser_limitations.txt` +- **Key Files**: `src/parser.c` (parser_parse_when_pattern), `tests/22_parser_limitations.txt` +- **Current Error**: `Parse error: Expected 'is' after test expression` +- **Working Test**: `echo "test_multi_expr : x y -> when (x % 2) (y % 2) is 0 0 then \"both even\";" | ./bin/baba-yaga` + +## Implementation Plan + +### **Phase 1: Core Language Features** ✅ **COMPLETE** +All core language features are now working correctly. + +### **Phase 2: Advanced Features** ✅ **COMPLETE** +All advanced features including partial application are now working. + +### **Phase 3: Final Polish** 🔄 **IN PROGRESS** + +#### **Task 3.1: Test 22 Parser Issue** (Test 22) 🔍 **INVESTIGATED** +**Issue**: `Parse error: Expected 'is' after test expression` +**Current**: Core multi-value pattern functionality works correctly +**Status**: Identified specific parser edge case - needs investigation + +**Investigation Findings**: +- ✅ **Individual functions work**: Multi-value patterns parse and execute correctly when tested individually +- ✅ **Isolated syntax works**: Same syntax works perfectly when tested via `echo` +- ❌ **File-specific issue**: The error only occurs when the complete test file is processed +- 🔍 **Parser edge case**: The issue appears to be in how the parser handles multiple patterns in sequence within a file context +- 📍 **Error location**: Parser fails to recognize the `is` keyword in multi-value pattern context when processing the full file + +**Root Cause Analysis**: +- The parser's `parser_parse_when_pattern` function may have an edge case when processing multiple patterns in sequence +- The error suggests the parser is not correctly transitioning between pattern parsing states +- This is likely a subtle parsing state management issue rather than a fundamental syntax problem + +#### **Task 3.2: Integration Test 02 File Reading** (Integration Test 02) +**Issue**: Segmentation fault when reading file directly (works when piped) +**Current**: Core pattern matching works, but file reading has issue +**Status**: Need to fix file reading mechanism + +## **Recent Achievements** + +### **Task 2.3: Partial Application Support** ✅ **COMPLETE** +- **Issue**: Test 17 failed with partial application and arity errors +- **Solution**: Implemented proper partial application in function call mechanism +- **Implementation**: + - Modified `baba_yaga_function_call` to handle partial application + - Created `stdlib_partial_apply` helper function + - Updated `each` function to support partial application +- **Result**: Test 17 now passes, 25/27 tests passing + +### **Task 1.2: Multi-value Pattern Expressions** ✅ **COMPLETE** +- **Issue**: `when (x % 2) (y % 2) is` not supported +- **Solution**: Enhanced parser to handle expressions in parentheses for multi-parameter patterns +- **Implementation**: Added detection for multi-parameter patterns with expressions +- **Result**: Multi-value pattern expressions now work correctly + +### **Task 1.3: Pattern Matching Memory** ✅ **COMPLETE** +- **Issue**: Segmentation fault in complex pattern matching +- **Solution**: Implemented sequence-to-sequence pattern matching for multi-parameter patterns +- **Implementation**: Added element-by-element comparison logic for multi-parameter patterns +- **Result**: Complex nested pattern matching now works correctly + +## **Next Priority** +**Task 3.1**: Fix Test 22 parser edge case to achieve 26/27 tests passing +**Task 3.2**: Fix Integration Test 02 file reading issue to achieve 27/27 tests passing + +## **Integration Test 02 Segfault Investigation** + +### Findings So Far +- Recursive function calls (e.g., `factorial 5`) cause a segmentation fault **only when run from a file**, not when piped via `cat` or `echo`. +- Non-recursive function calls, arithmetic, and function definitions all work as expected. +- The segfault occurs instantly, not after deep recursion (not a stack overflow). +- The function is defined in the global scope, and recursive lookup should work. +- The bug is **not** in the recursion logic itself. + +### Hypothesis +- The root cause is likely a memory or buffer issue in file reading, string handling, or tokenization. +- There may be a difference in how the source buffer is loaded from a file vs. piped input (e.g., BOM, encoding, or invisible characters). + +### Next Steps +1. Add debug output to print the raw contents of the buffer loaded by `read_file()` for `tests/integration_02_pattern_matching.txt` before it is passed to the interpreter. +2. Compare the buffer content from file vs. piped input. +3. Check for buffer overflows, uninitialized memory, or off-by-one errors in file reading and tokenization. +4. Check for non-ASCII, BOM, or invisible characters in the test file. + +--- + +## Technical Notes + +### **Partial Application Implementation** +- **Function Call Mechanism**: Modified `baba_yaga_function_call` to detect insufficient arguments +- **Partial Function Creation**: Creates new function with bound arguments stored in scope +- **Argument Combination**: `stdlib_partial_apply` combines bound and new arguments +- **Scope Management**: Uses temporary scope variables to store partial application data + +### **Pattern Matching Enhancements** +- **Multi-parameter Support**: Handles `when (expr1) (expr2) is` syntax +- **Sequence Comparison**: Element-by-element comparison for multi-value patterns +- **Wildcard Support**: `_` pattern matches any value in multi-parameter contexts + +### **Parser Investigation Results** +- **Multi-value patterns work correctly** in isolation and individual function definitions +- **File processing edge case** identified in `parser_parse_when_pattern` function +- **State management issue** suspected when processing multiple patterns in sequence +- **Error occurs specifically** when the complete test file is processed, not in isolated tests + +### **Memory Management** +- **Reference Counting**: Proper cleanup of function references +- **Scope Cleanup**: Automatic cleanup of temporary scope variables +- **Error Handling**: Graceful handling of memory allocation failures + +## Next Action +**Continue with Task 3.1** (Test 22 Parser Issue) - investigate and fix the parser edge case in `parser_parse_when_pattern` function to achieve 26/27 tests passing. + +## Implementation Guide + +### **For Task 3.1: Test 22 Parser Issue** +1. **Investigate `parser_parse_when_pattern` function**: Look for state management issues when processing multiple patterns +2. **Debug the specific failing case**: Add debug output to understand why the parser fails to recognize `is` keyword +3. **Fix the parser logic**: Update the parser to handle the edge case correctly +4. **Test the fix**: Verify that Test 22 now passes + +### **For Task 3.2: Integration Test 02 File Reading** +1. **Investigate the file reading issue**: Compare direct file reading vs piped input +2. **Identify the root cause**: Find why direct file reading causes segmentation fault +3. **Fix the file reading mechanism**: Update the file reading code to handle the issue +4. **Test the fix**: Verify that Integration Test 02 now passes + +### **For CLI Ergonomics** +1. **Simplify the REPL**: Make it more minimal and interactive +2. **Improve error messages**: Better error reporting and debugging +3. **Add helpful features**: Command history, line editing, etc. \ No newline at end of file |