diff options
Diffstat (limited to 'js/scripting-lang/baba-yaga-c/ROADMAP.md')
| -rw-r--r-- | js/scripting-lang/baba-yaga-c/ROADMAP.md | 911 |
1 files changed, 0 insertions, 911 deletions
diff --git a/js/scripting-lang/baba-yaga-c/ROADMAP.md b/js/scripting-lang/baba-yaga-c/ROADMAP.md deleted file mode 100644 index 87eb83f..0000000 --- a/js/scripting-lang/baba-yaga-c/ROADMAP.md +++ /dev/null @@ -1,911 +0,0 @@ -# Baba Yaga C Implementation Roadmap - -## Next Steps - Optional Polish - -1. **[OPTIONAL] Clean Up Debug Output:** Remove temporary debug printf statements from parser -2. **[RECOMMENDED] Comprehensive Test Sweep:** Run the full test suite to ensure no regressions -3. **[OPTIONAL] Performance Testing:** Test with larger recursive functions and complex expressions -4. **[OPTIONAL] Documentation:** Update README with recent fixes and improvements - ---- - -## Current Status - 🎉 COMPLETE! -- ✅ **Core Language**: Complete and stable -- ✅ **Table Pattern Matching**: Fixed and working -- ✅ **When Expressions**: Fixed and working -- ✅ **Computed Table Keys**: Fixed and working (Task 1.1 complete) -- ✅ **Multi-value Pattern Expressions**: Fixed and working (Task 1.2 complete) -- ✅ **Pattern Matching Memory**: Fixed and working (Task 1.3 complete) -- ✅ **Partial Application Support**: Fixed and working (Task 2.3 complete) -- ✅ **Test Runner**: Fixed to handle debug output properly -- ✅ **Function Reference in Call**: Fixed and working (Task 3.3 complete) -- ✅ **Debug System**: All debug output now properly controlled by DEBUG level -- ✅ **Parser Sequence Handling**: Fixed - now creates proper NODE_SEQUENCE nodes -- ✅ **Factorial Regression**: Fixed - `factorial 5` returns 120 correctly - -## Quick Reference -- **Test Command**: `./run_tests.sh` -- **Current Status**: All core functionality complete and working -- **Status**: Parser sequence handling fixed - recursive functions work perfectly -- **Debug Control**: Use `DEBUG=0-5` environment variable to control debug output -- **Build Command**: `make` -- **Key Files**: `src/interpreter.c`, `src/function.c`, `src/parser.c` - -## Project Setup and Structure - -### **Quick Start for Fresh Environment** -```bash -# Clone and build -git clone <repository> -cd baba-yaga-c -make - -# Run tests -./run_tests.sh - -# Run specific test -./bin/baba-yaga "add 5 @multiply 3 4" -``` - -### **Project Structure** -``` -baba-yaga-c/ -├── src/ # Core implementation -│ ├── main.c # Entry point, file I/O, debug setup -│ ├── lexer.c # Tokenization (source → tokens) -│ ├── parser.c # AST construction (tokens → AST) -│ ├── interpreter.c # AST evaluation (AST → values) -│ ├── function.c # Function call mechanism -│ ├── scope.c # Variable scope management -│ ├── value.c # Value type system -│ ├── table.c # Table data structure -│ ├── stdlib.c # Standard library functions -│ ├── debug.c # Debug logging system -│ └── memory.c # Memory management utilities -├── include/ # Header files -├── tests/ # Integration tests -├── bin/ # Compiled binary -├── run_tests.sh # Test runner script -└── Makefile # Build configuration -``` - -### **Key Components** -- **Lexer**: Converts source code to tokens (`lexer.c`) -- **Parser**: Builds Abstract Syntax Tree from tokens (`parser.c`) -- **Interpreter**: Evaluates AST to produce values (`interpreter.c`) -- **Function System**: Handles function calls and partial application (`function.c`) -- **Scope System**: Manages variable visibility and lifetime (`scope.c`) -- **Value System**: Type system for numbers, strings, booleans, functions (`value.c`) - -## Baba Yaga Language Semantics - -### **Core Language Features** - -#### **Basic Types and Values** -- **Numbers**: Integers and floating-point (`5`, `3.14`, `-2`) -- **Strings**: Text literals (`"hello"`, `"world"`) -- **Booleans**: `true` and `false` -- **Functions**: First-class function values -- **Tables**: Arrays and objects (see below) -- **Nil**: Null/undefined value - -#### **Variable Declarations and Assignment** -```baba-yaga -/* Variable declaration with assignment */ -x : 5; -name : "Alice"; -func : x -> x * 2; - -/* Multiple statements separated by semicolons */ -a : 1; b : 2; c : a + b; -``` - -#### **Arithmetic and Comparison Operators** -```baba-yaga -/* Arithmetic */ -sum : 5 + 3; /* Addition */ -diff : 10 - 4; /* Subtraction */ -product : 6 * 7; /* Multiplication */ -quotient : 15 / 3; /* Division */ -remainder : 17 % 5; /* Modulo */ - -/* Comparisons */ -is_equal : 5 = 5; /* Equality */ -is_less : 3 < 7; /* Less than */ -is_greater : 10 > 5; /* Greater than */ -is_less_equal : 5 <= 5; /* Less than or equal */ -is_greater_equal : 8 >= 8; /* Greater than or equal */ - -/* Logical operators */ -and_result : true and false; /* Logical AND */ -or_result : true or false; /* Logical OR */ -not_result : not false; /* Logical NOT */ -``` - -### **Functions** - -#### **Function Definition** -```baba-yaga -/* Basic function definition */ -add : x y -> x + y; -double : x -> x * 2; - -/* Recursive functions */ -factorial : n -> - when n is - 0 then 1 - _ then n * (factorial (n - 1)); -``` - -#### **Function Calls** -```baba-yaga -/* Direct function calls */ -result : add 5 3; -doubled : double 7; - -/* Function references with @ operator */ -add_ref : @add; -result2 : add_ref 10 20; -``` - -#### **Higher-Order Functions** -```baba-yaga -/* Function composition */ -composed : compose @double @square 3; - -/* Function piping */ -piped : pipe @double @square 2; - -/* Function application */ -applied : apply @double 7; - -/* Partial application (automatic) */ -add_five : add 5; /* Creates function that adds 5 */ -result3 : add_five 10; /* Result: 15 */ -``` - -### **Pattern Matching (Case Expressions)** - -#### **Basic Pattern Matching** -```baba-yaga -/* Single parameter patterns */ -grade : score -> - when score is - score >= 90 then "A" - score >= 80 then "B" - score >= 70 then "C" - _ then "F"; - -/* Wildcard patterns */ -factorial : n -> - when n is - 0 then 1 - _ then n * (factorial (n - 1)); -``` - -#### **Multi-Parameter Patterns** -```baba-yaga -/* Multiple parameter patterns */ -classify : x y -> - when x y is - 0 0 then "both zero" - 0 _ then "x is zero" - _ 0 then "y is zero" - _ _ then "neither zero"; - -/* Complex nested patterns */ -analyze : x y z -> - when x y z is - 0 0 0 then "all zero" - 0 0 _ then "x and y zero" - 0 _ 0 then "x and z zero" - _ 0 0 then "y and z zero" - 0 _ _ then "only x zero" - _ 0 _ then "only y zero" - _ _ 0 then "only z zero" - _ _ _ then "none zero"; -``` - -#### **Expression Patterns** -```baba-yaga -/* Patterns with expressions in parentheses */ -classify_parity : x y -> - when (x % 2) (y % 2) is - 0 0 then "both even" - 0 1 then "x even, y odd" - 1 0 then "x odd, y even" - 1 1 then "both odd"; -``` - -### **Tables (Arrays and Objects)** - -#### **Table Literals** -```baba-yaga -/* Empty table */ -empty : {}; - -/* Array-like table */ -numbers : {1, 2, 3, 4, 5}; - -/* Key-value table (object) */ -person : {name: "Alice", age: 30, active: true}; - -/* Mixed table (array + object) */ -mixed : {1, name: "Bob", 2, active: false}; -``` - -#### **Table Access** -```baba-yaga -/* Array access (1-indexed) */ -first : numbers[1]; -second : numbers[2]; - -/* Object access (dot notation) */ -name : person.name; -age : person.age; - -/* Object access (bracket notation) */ -name_bracket : person["name"]; -age_bracket : person["age"]; - -/* Mixed table access */ -first_mixed : mixed[1]; -name_mixed : mixed.name; -``` - -#### **Table Operations (t namespace)** -```baba-yaga -/* Immutable table operations */ -updated_person : t.set person "age" 31; -person_without_age : t.delete person "age"; -merged : t.merge person1 person2; - -/* Table utilities */ -length : t.length person; -has_name : t.has person "name"; -``` - -### **Table Combinators** - -#### **Map, Filter, Reduce** -```baba-yaga -/* Map with function */ -double : x -> x * 2; -doubled : map @double numbers; - -/* Filter with predicate */ -is_even : x -> x % 2 = 0; -evens : filter @is_even numbers; - -/* Reduce with accumulator */ -sum : x y -> x + y; -total : reduce @sum 0 numbers; -``` - -#### **Each Combinator** -```baba-yaga -/* Each for side effects */ -numbers : {1, 2, 3, 4, 5}; -each @print numbers; /* Prints each number */ -``` - -### **Input/Output Operations** - -#### **Output Commands** -```baba-yaga -/* Basic output */ -..out "Hello, World!"; - -/* Output with expressions */ -..out "Sum is: " + (5 + 3); -``` - -#### **Assertions** -```baba-yaga -/* Test assertions */ -..assert 5 + 3 = 8; -..assert factorial 5 = 120; -..assert person.name = "Alice"; -``` - -### **Language Characteristics** - -#### **Evaluation Strategy** -- **Eager Evaluation**: Arguments are evaluated immediately when assigned -- **First-Class Functions**: Functions can be passed as arguments, returned, and stored -- **Immutable Data**: Table operations return new tables, don't modify originals -- **Expression-Oriented**: Everything is an expression that produces a value - -#### **Scope and Binding** -- **Lexical Scoping**: Variables are bound in their defining scope -- **Function Scope**: Each function call creates a new local scope -- **Global Scope**: Variables defined at top level are globally accessible - -#### **Type System** -- **Dynamic Typing**: Types are determined at runtime -- **Type Coercion**: Automatic conversion between compatible types -- **Function Types**: Functions have arity (number of parameters) - -#### **Error Handling** -- **Graceful Degradation**: Invalid operations return nil or error values -- **Debug Output**: Extensive debug information available via DEBUG environment variable -- **Assertions**: Built-in assertion system for testing - -## Current Issue Details - -### **Failing Test: "Function Reference in Call"** -- **Test Expression**: `add 5 @multiply 3 4` -- **Expected Output**: `17` -- **Actual Output**: `Error: Execution failed` -- **Test Location**: `run_tests.sh` line 147 - -### **What This Test Does** -The test evaluates the expression `add 5 @multiply 3 4` which should: -1. Call `multiply` with arguments `3` and `4` (result: `12`) -2. Use `@` to reference the result as a function -3. Call `add` with arguments `5` and the result from step 1 (result: `17`) - -### **Investigation Context** -- **Function Reference Syntax**: The `@` operator creates a function reference -- **Nested Function Calls**: This tests calling a function with the result of another function call -- **Error Location**: The failure occurs during execution, not parsing -- **Related Issues**: May be connected to the parser precedence issues in Task 3.2 - -### **Debugging Approach** -```bash -# Test the failing expression directly -./bin/baba-yaga "add 5 @multiply 3 4" - -# Test components separately -./bin/baba-yaga "multiply 3 4" -./bin/baba-yaga "@multiply 3 4" -./bin/baba-yaga "add 5 12" - -# Run with debug output -DEBUG=4 ./bin/baba-yaga "add 5 @multiply 3 4" -``` - -## Implementation Plan - -### **Phase 1: Core Language Features** ✅ **COMPLETE** -All core language features are now working correctly. - -### **Phase 2: Advanced Features** ✅ **COMPLETE** -All advanced features including partial application are now working. - -### **Phase 3: Final Polish** ✅ **COMPLETE** - -#### **Task 3.3: Fix Function Reference in Call Test** ✅ **COMPLETE** -**Issue**: "Function Reference in Call" test fails with "Error: Execution failed" -**Solution**: Fixed parser to properly handle function references with arguments -**Implementation**: Modified `parser_parse_primary` to parse `@function args` as function calls -**Status**: 26/26 tests passing (100% completion) - -**Root Cause**: -- Parser was treating `@multiply` as a value rather than a function call -- `add 5 @multiply 3 4` was parsed as 4 arguments instead of nested function calls - -**Fix Applied**: -- Modified function reference parsing in `src/parser.c` -- Function references with arguments now create proper function call nodes -- Function references without arguments still return as values - -**Success Criteria**: -- ✅ Function Reference in Call test passes -- ✅ All 26 tests pass (100% completion) - -#### **Task 3.2: Integration Test 02 Parser Precedence Issue** 🔧 **INVESTIGATED** -**Root Cause Identified** ✅ **COMPLETE**: -- **Parser Precedence Bug**: The parser incorrectly interprets `factorial 3` as a binary operation `factorial - 3` (type 2) instead of a function call with literal argument (type 0) -- **AST Node Corruption**: Arguments are parsed as `NODE_BINARY_OP` instead of `NODE_LITERAL`, causing evaluation to produce corrupted values -- **Runtime Patch Applied**: Interpreter-level fix attempts to detect and correct corrupted arguments -- **Status**: This issue was investigated but is not currently blocking test completion - -**Current Status**: -- ❌ **Simple Function Calls**: `factorial 3` still parses as `NODE_BINARY_OP` argument (type 2) -- ❌ **Runtime Patch**: Interpreter detects corruption but cannot prevent segfault -- ❌ **Function Execution**: Both `test_var_decl_call.txt` and integration test segfault -- ❌ **Complex Expressions**: Variable declarations with function calls still parse arguments as `NODE_BINARY_OP` -- ❌ **Integration Test**: Full test still segfaults despite runtime patch - -**Implementation Plan**: - -**Step 1: Fix Parser Precedence** 🔧 **PENDING** -- **Issue**: Function application has lower precedence than binary operations -- **Fix**: Restructure parser to give function application highest precedence -- **Files**: `src/parser.c` - `parser_parse_expression()`, `parser_parse_application()` -- **Test**: Verify `fact5 : factorial 5;` parses argument as `NODE_LITERAL` (type 0) - -**Step 2: Remove Runtime Patch** 🔧 **PENDING** -- **Issue**: Runtime patch masks underlying parser bug -- **Fix**: Remove interpreter-level corruption detection and fix -- **Files**: `src/interpreter.c` - `interpreter_evaluate_expression()` case `NODE_FUNCTION_CALL` -- **Test**: Verify function calls work without runtime intervention - -**Step 3: Integration Test Validation** ✅ **PENDING** -- **Test**: Run `tests/integration_02_pattern_matching.txt` successfully -- **Expected**: No segfault, correct output for all assertions -- **Validation**: All 26 tests should pass (currently 25/26) - -**Success Criteria**: -- ✅ Integration Test 02 passes without segfault -- ✅ Function call arguments parse as `NODE_LITERAL` (type 0) -- ✅ No runtime patches needed for argument corruption -- ✅ All 26 tests pass (100% completion) - -#### **Task 3.1: Test 22 Parser Issue** (Test 22) 🔍 **INVESTIGATED** -**Issue**: `Parse error: Expected 'is' after test expression` -**Current**: Core multi-value pattern functionality works correctly -**Status**: Identified specific parser edge case - needs investigation - -**Investigation Findings**: -- ✅ **Individual functions work**: Multi-value patterns parse and execute correctly when tested individually -- ✅ **Isolated syntax works**: Same syntax works perfectly when tested via `echo` -- ❌ **File-specific issue**: The error only occurs when the complete test file is processed -- 🔍 **Parser edge case**: The issue appears to be in how the parser handles multiple patterns in sequence within a file context -- 📍 **Error location**: Parser fails to recognize the `is` keyword in multi-value pattern context when processing the full file - -**Root Cause Analysis**: -- The parser's `parser_parse_when_pattern` function may have an edge case when processing multiple patterns in sequence -- The error suggests the parser is not correctly transitioning between pattern parsing states -- This is likely a subtle parsing state management issue rather than a fundamental syntax problem - -## **Recent Achievements** - -### **Function Reference in Call Fix** ✅ **COMPLETE** -- **Issue**: "Function Reference in Call" test failed with "Error: Execution failed" -- **Root Cause**: Parser treated `@multiply` as a value instead of a function call -- **Solution**: Modified `parser_parse_primary` to parse function references with arguments as function calls -- **Implementation**: Updated function reference parsing logic in `src/parser.c` -- **Result**: All 26 tests pass (100% completion) - -### **Test Runner Fix** ✅ **COMPLETE** -- **Issue**: Test runner was failing because debug output was mixed with test results -- **Solution**: Patched `run_tests.sh` to filter out `DEBUG:` lines before comparing outputs -- **Implementation**: Added `grep -v '^DEBUG:'` to the `run_simple_test()` function -- **Result**: Now 26/26 tests pass (100% completion) - -### **Parser Precedence Investigation** ✅ **COMPLETE** -- **Systematic Approach**: Used isolated test cases to identify parser behavior - - Simple function call: ❌ Fails (`factorial 3` → `NODE_BINARY_OP` argument) - - Variable declaration with function call: ❌ Fails (`fact5 : factorial 5;` → `NODE_BINARY_OP` argument) - - Complex integration test: ❌ Fails (mixed parsing behavior) -- **Root Cause Isolation**: Identified parser precedence as the bottleneck -- **Evidence-Based Diagnosis**: Used debug output to trace AST node types -- **Runtime Patch Implementation**: Created temporary fix to attempt function execution - -### **Runtime Patch Implementation** ✅ **COMPLETE** -- **Deep Copy Logic**: Implemented proper argument value copying to prevent corruption -- **Validation System**: Added argument type and value validation after copying -- **Corruption Detection**: Automatic detection of negative argument values (indicating corruption) -- **Automatic Fix**: Runtime correction of corrupted arguments using default values -- **Function Execution**: Attempts to allow `factorial` function to execute but still segfaults - -### **JS Team Consultation** ✅ **COMPLETE** -- **Consultation**: Received comprehensive response from Baba Yaga JS implementation team -- **Key Insights**: - - **Immediate Evaluation**: Arguments must be evaluated immediately when assignments are processed - - **Memory Safety**: Proper argument array allocation and preservation required - - **Scope Management**: Fresh local scope needed for each recursive call - - **No File vs Pipe Differences**: Both input methods should work identically -- **Impact**: Confirmed that parser precedence is the correct focus area - -### **Task 2.3: Partial Application Support** ✅ **COMPLETE** -- **Issue**: Test 17 failed with partial application and arity errors -- **Solution**: Implemented proper partial application in function call mechanism -- **Implementation**: - - Modified `baba_yaga_function_call` to handle partial application - - Created `stdlib_partial_apply` helper function - - Updated `each` function to support partial application -- **Result**: Test 17 now passes, 25/26 tests passing - -### **Task 1.2: Multi-value Pattern Expressions** ✅ **COMPLETE** -- **Issue**: `when (x % 2) (y % 2) is` not supported -- **Solution**: Enhanced parser to handle expressions in parentheses for multi-parameter patterns -- **Implementation**: Added detection for multi-parameter patterns with expressions -- **Result**: Multi-value pattern expressions now work correctly - -### **Task 1.3: Pattern Matching Memory** ✅ **COMPLETE** -- **Issue**: Segmentation fault in complex pattern matching -- **Solution**: Implemented sequence-to-sequence pattern matching for multi-parameter patterns -- **Implementation**: Added element-by-element comparison logic for multi-parameter patterns -- **Result**: Complex nested pattern matching now works correctly - -## Recent Achievements - -### REPL Function Call Fix (Latest) -- **Issue**: Functions defined in REPL couldn't be called in subsequent lines -- **Root Cause**: AST nodes for function bodies were destroyed after each REPL execution, leaving dangling pointers -- **Solution**: Implemented deep AST node copying (`ast_copy_node`) to preserve function bodies -- **Implementation**: - - Added `ast_copy_node()` function in `src/parser.c` with support for common node types - - Modified function creation in `src/interpreter.c` to copy AST nodes instead of storing direct pointers - - Handles `NODE_LITERAL`, `NODE_IDENTIFIER`, `NODE_BINARY_OP`, `NODE_UNARY_OP`, `NODE_FUNCTION_CALL`, `NODE_WHEN_EXPR`, `NODE_WHEN_PATTERN` -- **Results**: - - ✅ Simple functions work: `f : x -> x + 1; f 5` returns `6` - - ✅ Recursive functions work: `factorial 5` returns `120` - - ✅ Multi-parameter functions work: `add : x y -> x + y; add 3 4` returns `7` - - ✅ Partial application works: `partial : add 10; partial 5` returns `15` -- **Files Modified**: `src/parser.c` (AST copy), `src/interpreter.c` (function creation) - -### Test Runner Implementation -- **Enhancement**: Implemented C-based test runner with `-t` flag -- **Features**: - - Automatic discovery of `.txt` test files in directory - - Execution of test code with error handling - - Beautiful output with ✅/❌ status indicators - - Comprehensive test summary with pass/fail counts - - Integration with `make test` command -- **Results**: 25/34 tests passing (74% success rate) -- **Usage**: `./bin/baba-yaga -t tests/` or `make test` -- **Files Modified**: `src/main.c` (test runner implementation), `Makefile` (test target) - -### Enhanced REPL + IO Namespace Fix -- **Enhancement**: Added interactive REPL mode with `--repl` flag -- **Features**: - - Beautiful interface with `🧙♀️ Baba Yaga Interactive REPL` header - - Built-in commands: `help`, `clear`, `exit`/`quit` - - Enhanced output with `=>` prefix for results - - Friendly error messages with visual indicators -- **Pipe-Friendly**: Default behavior reads from stdin (perfect for scripts and pipes) -- **IO Namespace Fix**: Corrected documentation to use proper `..out`, `..in`, `..listen`, `..emit` syntax -- **Backward Compatibility**: All existing functionality preserved -- **Files Modified**: `src/main.c` (command-line interface and REPL implementation) - -### Parser Sequence Handling Fix -- **Problem**: Parser was not creating proper `NODE_SEQUENCE` nodes for multiple statements -- **Symptoms**: - - Simple sequences worked: `x : 1; y : 2;` - - Function + statement sequences failed: `factorial : n -> ...; factorial 5;` - - Recursive functions like `factorial 5` returned errors instead of results -- **Root Cause**: `parser_parse_when_result_expression` was calling `parser_parse_primary()` instead of `parser_parse_expression()`, preventing complex expressions like `countdown (n - 1)` from being parsed correctly -- **Solution**: - - Changed `parser_parse_primary(parser)` to `parser_parse_expression(parser)` in when expression result parsing - - Removed semicolon consumption from function definition parser (let statement parser handle it) -- **Result**: - - Parser now creates proper `NODE_SEQUENCE` nodes for multiple statements - - `factorial 5` returns `120` correctly - - All recursive functions work perfectly -- **Files Modified**: `src/parser.c` (lines 2776, 1900-1904) - -### Function Reference in Call Fix -- **Problem**: `add 5 @multiply 3 4` was parsed as `add(5, @multiply, 3, 4)` instead of `add(5, multiply(3, 4))` -- **Root Cause**: Parser was explicitly treating function references as values, not function calls -- **Solution**: Modified `parser_parse_primary()` to correctly parse `@function args` as function calls -- **Result**: Function reference in call test now passes (Task 3.3 complete) - -### Debug System Cleanup -- **Problem**: Debug output not respecting `DEBUG=0` environment variable -- **Root Cause**: Hardcoded `printf("DEBUG: ...")` statements instead of using debug macros -- **Solution**: Replaced all hardcoded debug prints with `DEBUG_DEBUG`, `DEBUG_INFO`, `DEBUG_WARN` macros -- **Files Fixed**: `src/interpreter.c`, `src/main.c`, `src/function.c` -- **Result**: All debug output now properly controlled by DEBUG level - ---- - -## Factorial Regression Investigation - -### Initial Problem Discovery -- **Issue**: `factorial 5` was returning "Error: Execution failed" instead of 120 -- **Context**: This was discovered during debug system cleanup testing -- **Impact**: Blocked 100% test completion - -### Investigation Process - -#### Phase 1: Debug Output Analysis -- **Method**: Used `DEBUG=5` to trace execution -- **Findings**: - - Function was calling itself infinitely - - Corruption detection was triggering on negative values (-1) - - Segmentation fault due to stack overflow - -#### Phase 2: Corruption Detection Logic -- **Location**: `src/interpreter.c` lines 585-593 -- **Problem**: Interpreter was treating negative values as corruption and "fixing" them to 3 -- **Impact**: Created infinite loop: `factorial(3)` → `factorial(2)` → `factorial(1)` → `factorial(0)` → `factorial(-1)` → `factorial(3)` (corruption fix) → repeat -- **Solution**: Removed corruption detection logic -- **Result**: Eliminated infinite loop, but factorial still failed - -#### Phase 3: Parser Sequence Issue Discovery -- **Method**: Tested different statement sequences -- **Findings**: - - Simple sequences work: `x : 1; y : 2;` ✅ - - Function + statement sequences fail: `factorial : n -> ...; factorial 5;` ❌ - - Parser creates wrong node types: - - Expected: `NODE_SEQUENCE` (type 13) - - Actual: `NODE_FUNCTION_DEF` (type 5) for factorial case - - Actual: `NODE_TABLE_ACCESS` (type 12) for simple sequences - -#### Phase 4: Root Cause Analysis -- **Problem**: Parser not properly handling semicolon-separated statements -- **Location**: `parser_parse_statements()` in `src/parser.c` -- **Issue**: Parser stops after parsing function definition, doesn't continue to parse semicolon and next statement -- **Impact**: Only first statement is executed, subsequent statements are ignored - -### Technical Details - -#### Corruption Detection Logic (Removed) -```c -// REMOVED: This was causing infinite loops -if (args[i].type == VAL_NUMBER && args[i].data.number < 0) { - DEBUG_WARN("First argument is negative (%g), this indicates corruption!", - args[i].data.number); - DEBUG_DEBUG("Attempting to fix corruption by using default value 3"); - args[i] = baba_yaga_value_number(3); -} -``` - -#### Parser Sequence Issue -- **Function**: `parser_parse_statements()` in `src/parser.c` lines 1972-2070 -- **Expected Behavior**: Create `NODE_SEQUENCE` when multiple statements found -- **Actual Behavior**: Returns only first statement, ignores semicolon and subsequent statements -- **Debug Evidence**: - - Simple sequence: `Evaluating expression: type 12` (should be 13) - - Factorial case: `Evaluating expression: type 5` (NODE_FUNCTION_DEF) - -#### Debug System Fixes Applied -- **Files Modified**: `src/interpreter.c`, `src/main.c`, `src/function.c` -- **Changes**: Replaced `printf("DEBUG: ...")` with `DEBUG_DEBUG("...")` -- **Result**: Debug output now properly respects `DEBUG` environment variable - -### Current Status -- ✅ **Debug System**: Fully functional and properly controlled -- ❌ **Parser Sequence Handling**: Not creating proper NODE_SEQUENCE nodes -- ❌ **Factorial Regression**: Still failing due to parser issue -- 🔍 **Root Cause**: Parser stops after function definition, doesn't parse subsequent statements - -### Next Steps -1. **Fix Parser Sequence Handling**: Modify `parser_parse_statements()` to properly create sequence nodes -2. **Test Factorial**: Verify factorial works after parser fix -3. **Run Full Test Suite**: Ensure no other regressions -4. **Update Documentation**: Reflect all fixes in README - -### Lessons Learned -- **Debug System**: Always use proper debug macros, not hardcoded prints -- **Parser Testing**: Test edge cases like function + statement sequences -- **Corruption Detection**: Be careful with "fixes" that mask real bugs -- **Investigation Process**: Use systematic debugging to isolate root causes - ---- - -## Next Priority - -**[COMPLETE] All Core Functionality Working** -- ✅ Parser sequence handling: Fixed - now creates proper NODE_SEQUENCE nodes -- ✅ Factorial regression: Fixed - `factorial 5` returns 120 correctly -- ✅ Debug system cleanup: Complete - all debug output macro-controlled -- ✅ Function reference in calls: Fixed and working - -**[OPTIONAL] Remaining Tasks** -All remaining tasks are optional polish: -- ✅ **Documentation Updated**: Comprehensive README with language guide, semantics, and development info -- ✅ **Test Runner Implemented**: C-based test runner with 25/34 tests passing -- Investigate and fix failing tests (advanced features like embedded functions, function composition) -- Clean up any remaining temporary debug statements -- Performance testing with larger recursive functions - -## Technical Notes - -### **Parser Precedence Implementation** -- **Function Application**: Should have highest precedence in expression parsing -- **Current Issue**: Function application handled at lower precedence than binary operations -- **Solution**: Restructure `parser_parse_expression()` to call `parser_parse_application()` first -- **Expected Result**: All function call arguments parse as `NODE_LITERAL` (type 0) -- **Current Status**: Parser precedence fix not working - arguments still parsed as `NODE_BINARY_OP` - -### **Runtime Patch Details** -- **Deep Copy**: Proper copying of `Value` types to prevent corruption -- **Validation**: Type and value checking after argument copying -- **Corruption Detection**: Automatic detection of negative numbers in function arguments -- **Automatic Fix**: Runtime correction using default values (e.g., `3` for `factorial`) -- **Temporary Nature**: This patch masks the underlying parser bug and should be removed -- **Current Status**: Patch detects corruption but cannot prevent segfault - -### **Partial Application Implementation** -- **Function Call Mechanism**: Modified `baba_yaga_function_call` to detect insufficient arguments -- **Partial Function Creation**: Creates new function with bound arguments stored in scope -- **Argument Combination**: `stdlib_partial_apply` combines bound and new arguments -- **Scope Management**: Uses temporary scope variables to store partial application data - -### **Pattern Matching Enhancements** -- **Multi-parameter Support**: Handles `when (expr1) (expr2) is` syntax -- **Sequence Comparison**: Element-by-element comparison for multi-value patterns -- **Wildcard Support**: `_` pattern matches any value in multi-parameter contexts - -### **Memory Management** -- **Reference Counting**: Proper cleanup of function references -- **Scope Cleanup**: Automatic cleanup of temporary scope variables -- **Error Handling**: Graceful handling of memory allocation failures - -## Next Action -**🎉 Implementation Complete + Parser Fixed!** - -The Baba Yaga C implementation is now fully functional with all critical issues resolved: -- Parser sequence handling works correctly -- Recursive functions like `factorial 5` work perfectly -- Debug system is properly controlled -- All core functionality is stable and working - -**Optional next steps**: Clean up debug output, run comprehensive tests, performance testing. - -## Test Output and Debug Logging: Best Practices - -- For reliable automated testing, **all debug and diagnostic output should go to stderr**. -- Only the final program result (the value to be tested) should be printed to stdout. -- This ensures that test runners and scripts can compare outputs directly without filtering. -- **Current workaround:** The test runner is patched to filter out lines starting with 'DEBUG:' before comparing outputs, so tests can pass even with debug output present. -- **Long-term solution:** Refactor the C code so that all debug output uses `fprintf(stderr, ...)` or the project's debug logging macros, and only results are printed to stdout. -- This will make the codebase more portable, easier to test, and more robust for CI/CD and future contributors. - -## Debug System Cleanup Plan - -### Current State Analysis -- **Existing Infrastructure:** There's already a proper debug system with environment variable control (`DEBUG=0-5`) -- **Mixed Implementation:** Some code uses the debug macros (`DEBUG_ERROR`, `DEBUG_DEBUG`, etc.), but most uses hardcoded `printf("DEBUG: ...")` statements -- **Inconsistent Output:** Debug output goes to both stdout and stderr, causing test failures - -### Debug Levels Available -- `DEBUG_NONE = 0` - No debug output -- `DEBUG_ERROR = 1` - Only errors -- `DEBUG_WARN = 2` - Warnings and errors -- `DEBUG_INFO = 3` - Info, warnings, and errors -- `DEBUG_DEBUG = 4` - Debug, info, warnings, and errors -- `DEBUG_TRACE = 5` - All debug output - -### Cleanup Plan - -#### Phase 1: Replace Hardcoded Debug Output (Priority: High) -1. **Replace all `printf("DEBUG: ...")` with `fprintf(stderr, "DEBUG: ...")`** - - Files: `src/interpreter.c`, `src/function.c`, `src/main.c` - - This ensures debug output goes to stderr and doesn't interfere with test results - -2. **Replace `printf("DEBUG: ...")` with proper debug macros** - - Use `DEBUG_DEBUG()` for general debug info - - Use `DEBUG_TRACE()` for detailed execution tracing - - Use `DEBUG_ERROR()` for error conditions - -#### Phase 2: Implement Conditional Debug Output (Priority: Medium) -1. **Wrap debug output in debug level checks** - ```c - if (interp->debug_level >= DEBUG_DEBUG) { - fprintf(stderr, "DEBUG: Processing NODE_LITERAL\n"); - } - ``` - -2. **Use debug macros consistently** - ```c - DEBUG_DEBUG("Processing NODE_LITERAL"); - DEBUG_TRACE("Binary operator: %s", operator); - ``` - -#### Phase 3: Remove Test Runner Filtering (Priority: Low) -1. **Once all debug output is properly controlled, remove the `grep -v '^DEBUG:'` filter from the test runner** -2. **Set `DEBUG=0` in test environment to suppress all debug output** - -### Implementation Steps - -#### Step 1: Quick Fix (Immediate) -- Replace all remaining `printf("DEBUG: ...")` with `fprintf(stderr, "DEBUG: ...")` -- This fixes test failures immediately - -#### Step 2: Proper Debug Control (Next) -- Wrap debug output in `if (interp->debug_level >= DEBUG_DEBUG)` checks -- Use debug macros where appropriate - -#### Step 3: Clean Test Environment (Final) -- Set `DEBUG=0` in test runner -- Remove debug filtering from test runner -- Ensure clean test output - -### Usage Examples -```bash -# No debug output (default) -./bin/baba-yaga "5 + 3;" - -# Show debug output -DEBUG=4 ./bin/baba-yaga "5 + 3;" - -# Show all trace output -DEBUG=5 ./bin/baba-yaga "5 + 3;" - -# Run tests with no debug output -DEBUG=0 ./run_tests.sh -``` - -## Troubleshooting Guide - -### **Common Issues When Starting Fresh** - -#### **Build Issues** -```bash -# If make fails, try: -make clean -make - -# If still failing, check dependencies: -# - GCC compiler -# - Make utility -# - Standard C libraries -``` - -#### **Test Runner Issues** -```bash -# If tests show many failures, check: -./run_tests.sh | grep -A 5 -B 5 "FAIL" - -# If debug output is mixed with results: -# The test runner should filter this automatically -# If not, check that run_tests.sh contains the grep filter -``` - -#### **Segmentation Faults** -```bash -# If you get segfaults, run with debug: -DEBUG=4 ./bin/baba-yaga "your_expression" - -# Common segfault locations: -# - src/interpreter.c: NODE_FUNCTION_CALL case -# - src/function.c: baba_yaga_function_call -# - src/parser.c: parser_parse_expression -``` - -#### **Parser Issues** -```bash -# Test parser behavior: -./bin/baba-yaga "factorial 3" -./bin/baba-yaga "fact5 : factorial 5;" - -# Look for "NODE_BINARY_OP" in debug output (indicates parser precedence issue) -``` - -#### **Function Reference Issues** -```bash -# Test function reference syntax: -./bin/baba-yaga "@multiply" -./bin/baba-yaga "add 5 @multiply 3 4" - -# Check if @ operator is working correctly -``` - -### **Debug Output Interpretation** - -#### **AST Node Types** -- `type 0`: `NODE_LITERAL` (correct for function arguments) -- `type 2`: `NODE_BINARY_OP` (incorrect - indicates parser precedence issue) -- `type 3`: `NODE_FUNCTION_CALL` -- `type 4`: `NODE_IDENTIFIER` - -#### **Common Debug Messages** -- `"DEBUG: Processing NODE_LITERAL"` - Normal execution -- `"DEBUG: Processing NODE_BINARY_OP"` - May indicate parser issue -- `"WARNING: First argument is negative"` - Indicates argument corruption -- `"DEBUG: Function call arg_count"` - Function call processing - -### **Investigation Workflow** -1. **Reproduce the issue** with the exact failing expression -2. **Test components separately** to isolate the problem -3. **Check debug output** for AST node types and execution flow -4. **Compare with working cases** to identify differences -5. **Focus on the specific failing component** (parser, interpreter, function system) - -### **Key Files for Common Issues** -- **Parser Issues**: `src/parser.c` - `parser_parse_expression()`, `parser_parse_application()` -- **Function Call Issues**: `src/function.c` - `baba_yaga_function_call()` -- **Interpreter Issues**: `src/interpreter.c` - `interpreter_evaluate_expression()` -- **Scope Issues**: `src/scope.c` - `scope_get()`, `scope_set()` -- **Value Issues**: `src/value.c` - `value_copy()`, `value_destroy()` - -### **Environment Variables** -```bash -# Debug levels -DEBUG=0 # No debug output -DEBUG=1 # Errors only -DEBUG=2 # Warnings and errors -DEBUG=3 # Info, warnings, and errors -DEBUG=4 # Debug, info, warnings, and errors -DEBUG=5 # All debug output (trace) - -# Examples -DEBUG=4 ./bin/baba-yaga "add 5 @multiply 3 4" -DEBUG=0 ./run_tests.sh -``` |