path: root/js/scripting-lang/NEXT-STEPS.md

# Next Steps: Immutable Real-World Programming Features

## Overview

This document outlines the plan for extending the Simple Scripting Language to support real-world programming scenarios while maintaining its immutable, functional design philosophy.

## Core Principles

- **Immutability First**: All data structures are immutable
- **Transformation Over Mutation**: Operations return new data rather than modifying existing data
- **Functional Composition**: Complex operations built from simple, composable functions
- **Type Safety**: Enhanced pattern matching and type checking
- **Performance**: Efficient persistent data structures

## Phase 1: String Operations and Type System

### 1.1 String Operations
**Goal**: Add essential string manipulation capabilities

**New Functions**:
```javascript
// String operations as functions
length : string.length "hello";           // 5
contains : string.contains "hello" "ll";  // true
startsWith : string.startsWith "hello" "he"; // true
endsWith : string.endsWith "hello" "lo";  // true
substring : string.substring "hello" 1 3; // "el"

// String concatenation
result : string.concat "Hello" " " "World"; // "Hello World"

// String transformation
uppercase : string.upper "hello";         // "HELLO"
lowercase : string.lower "HELLO";         // "hello"
trimmed : string.trim "  hello  ";        // "hello"
```

**Implementation**:
- Add `string.` namespace for string operations
- Add string operation functions to standard library
- No new syntax - uses existing function call patterns
- Extend existing string literal support

### 1.2 Runtime Type Checking with `is` Keyword
**Goal**: Add explicit, optional type checking mechanism using the `is` keyword

**Design Philosophy**:
- Type checking is purely additive - no breaking changes to existing code
- Works seamlessly with existing case expression syntax
- Returns boolean values, fitting the functional style
- Simple implementation with clear semantics

**New Features**:
```javascript
// Basic type checking
isNumber : x -> x is number;
isString : x -> x is string;
isTable : x -> x is table;
isFunction : x -> x is function;

// Type-safe operations in case expressions
safeStringOp : x -> case x of
    x is string : string.length x
    _ : "not a string";

// Complex type validation
validateUser : user -> case user of
    user.name is string and user.age is number : true
    _ : false;

// Type guards in pattern matching
processData : data -> case data of
    data is table : "processing table"
    data is string : "processing string"
    data is number : "processing number"
    _ : "unknown type";
```

**Supported Types**:
- `number` (both integers and floats)
- `string` 
- `boolean`
- `table` (objects and arrays)
- `function`

**Implementation**:
- Add `IS` token type and type-specific tokens (`NUMBER_TYPE`, `STRING_TYPE`, etc.)
- Update lexer to recognize `is` keyword and type names
- Extend parser to handle type checking expressions in case patterns
- Add type checking logic in interpreter that returns boolean values
- No changes to existing syntax or semantics

## Phase 2: Persistent Data Structures

### 2.1 Immutable Tables with Transformations
**Goal**: Replace mutable table operations with immutable transformations

**Current Problem**:
```javascript
// This won't work (mutable)
cache[key] = value;
```

**Solution - Functional Immutable Transformations**:
```javascript
// Functional transformations using table namespace
cache : {};
cache1 : table.set cache "user.1" "Alice";
cache2 : table.set cache1 "user.2" "Bob";
value : table.get cache2 "user.1";  // "Alice"
cache3 : table.delete cache2 "user.1";

// Nested updates with dot notation
user : {name: "Alice", profile: {age: 25}};
updatedUser : table.set user "profile.age" 26;

// Complex transformations using composition
cache1 : pipe
    (table.set "user.1" "Alice")
    (table.set "user.2" "Bob")
    (table.set "user.3" "Charlie")
    cache;

// Table merging
combined : table.merge table1 table2;
```

**Implementation**:
- Add `table.set`, `table.get`, `table.delete`, `table.merge` functions
- Implement efficient structural sharing for immutable updates
- Add nested key support (e.g., "profile.age")
- All functions return new tables, never modify originals
- Use existing function composition patterns

### 2.2 APL-Inspired Table Primitives
**Goal**: Enhance tables with APL-style operations for ergonomic data manipulation

**Design Philosophy**:
- Use existing table syntax for array-like behavior
- Add APL-inspired primitives for vectorized operations
- Keep immutable transformations
- Provide concise, expressive data manipulation

**New Features**:
```javascript
// Table creation (array-like using existing syntax)
numbers : {1, 2, 3, 4, 5};
mixed : {1, "hello", true, {key: "value"}};

// All table operations namespaced for consistency
first : table.first numbers;              // First element
last : table.last numbers;                // Last element
length : table.length numbers;            // Size/shape

// Less common operations (namespaced)
rest : table.drop 1 numbers;        // Drop first element  
slice : table.slice numbers 2 3;    // Elements 2-3

// Vectorized operations
doubled : table.add numbers numbers;      // Element-wise addition
squared : table.multiply numbers numbers; // Element-wise multiplication
incremented : table.add numbers 1;        // Scalar addition to each element

// Reductions (all namespaced)
sum : table.sum numbers;                 // Sum reduction
max : table.max numbers;                 // Maximum reduction
min : table.min numbers;                 // Minimum reduction
product : table.product numbers;   // Product reduction
average : table.average numbers;   // Average reduction

// Scan operations (namespaced)
runningSum : table.scan table.sum numbers;        // Running sum scan
runningProduct : table.scan table.product numbers; // Running product scan

// Table metadata operations
keys : table.keys table;           // Get all keys
values : table.values table;       // Get all values
reversed : table.reverse table;    // Reverse order
sorted : table.sort table;         // Sort
unique : table.unique table;       // Remove duplicates
```

**Implementation**:
- Add `table.` namespace for all table operations
- Extend lexer to recognize table operation keywords
- Add vectorized operation logic in interpreter
- Implement reduction and scan operations
- Add table metadata operations

## Phase 3: Higher-Order Functions for Collections

### 3.1 Enhanced Standard Library
**Goal**: Add collection processing functions

**New Functions**:
```javascript
// Table processing (using namespaced primitives)
numbers : {1, 2, 3, 4, 5};
doubled : map @double numbers;                // {2, 4, 6, 8, 10}
evens : filter @isEven numbers;               // {2, 4}
sum : table.sum numbers;                      // 15 (sum reduction)

// Table metadata operations
table : {a: 1, b: 2, c: 3};
keys : table.keys table;                      // {a, b, c}
values : table.values table;                  // {1, 2, 3}
pairs : table.pairs table;                    // {{a, 1}, {b, 2}, {c, 3}}

// Advanced operations with tables
nested : {{1, 2}, {3, 4}, {5}};
flattened : table.flatten nested;             // {1, 2, 3, 4, 5}
grouped : table.groupBy @isEven numbers;     // {true: {2, 4}, false: {1, 3, 5}}

// Table operations
reversed : table.reverse numbers;             // {5, 4, 3, 2, 1}
sorted : table.sort numbers;                  // {1, 2, 3, 4, 5}
unique : table.unique {1, 2, 2, 3, 3, 4};   // {1, 2, 3, 4}
```

**Implementation**:
- Extend existing `map`, `filter`, `reduce` to work with tables
- Implement vectorized operations for tables
- Add reduction and scan operations
- Implement table metadata operations

### 3.2 Table Generation Helpers
**Goal**: Add convenient table creation functions

**New Functions**:
```javascript
// Table generation helpers
range : table.range 1 5;           // {1, 2, 3, 4, 5}
repeated : table.repeat "hello" 3; // {"hello", "hello", "hello"}

// Use existing map/filter instead of comprehensions
squares : map (x -> x * x) {1, 2, 3, 4, 5};
evens : filter @isEven {1, 2, 3, 4, 5};
```

## Phase 4: Error Handling with Error Type

### 4.1 Error Type
**Goal**: Provide consistent error handling without complex monads

**Implementation**:
```javascript
// Simple error type
error : message -> {type: "error", message: message};

// Safe operations with error handling
safeDivide : x y -> case y of
    0 : error "division by zero"
    _ : x / y;

safeParseNumber : str -> case str of
    str is number : str
    _ : error "invalid number";

// Error checking
isError : value -> value is error;
getErrorMessage : error -> case error of
    {type: "error", message: m} : m;
```

## Phase 5: Real-World Scenario Support

### 5.1 User Management System
**Immutable Implementation**:
```javascript
// User validation
isValidEmail : email -> case email of
    email contains "@" : true
    _ : false;

createUser : name email age -> case (isValidEmail email) and (isValidAge age) of
    true : {name: name, email: email, age: age, status: "active"}
    false : error "invalid user data";

// User management
users : {};
user1 : createUser "Alice" "alice@example.com" 25;
users1 : table.set users "user.1" user1;
user2 : createUser "Bob" "bob@example.com" 30;
users2 : table.set users1 "user.2" user2;

// Safe user lookup
findUser : email users -> case users of
    {} : error "user not found"
    _ : case (table.first users).email = email of
        true : table.first users
        false : findUser email (table.drop 1 users);
```

### 5.2 Shopping Cart System
**Immutable Implementation**:
```javascript
// Cart operations
emptyCart : {items: {}, total: 0};
addItem : cart item -> {
    items: table.set cart.items (table.length cart.items + 1) item,
    total: cart.total + item.price
};
removeItem : cart itemId -> {
    items: filter (item -> item.id != itemId) cart.items,
    total: calculateTotal (filter (item -> item.id != itemId) cart.items)
};

// Discount application
applyDiscount : cart discountPercent -> {
    items: cart.items,
    total: cart.total * (1 - discountPercent / 100)
};
```

### 5.3 Data Processing Pipeline
**Immutable Implementation**:
```javascript
// Data processing
salesData : {
    {month: "Jan", sales: 1000, region: "North"},
    {month: "Feb", sales: 1200, region: "North"},
    {month: "Mar", sales: 800, region: "South"}
};

// Filter by region
filterByRegion : data region -> filter (item -> item.region = region) data;

// Calculate totals using sum reduction
sumSales : data -> table.sum (map (item -> item.sales) data);

// Process pipeline
northData : filterByRegion salesData "North";
northTotal : sumSales northData;
```

## Implementation Timeline

### Week 1-2: String Operations and Runtime Type Checking
- [ ] String concatenation operator
- [ ] String method implementations
- [ ] `is` keyword and type checking tokens
- [ ] Type checking in case expressions
- [ ] Type validation functions in standard library

### Week 3-4: Table Primitives with Namespacing
- [ ] `table.` namespace for all table operations
- [ ] Vectorized operations for tables
- [ ] Reduction and scan operations
- [ ] Table metadata operations
- [ ] Performance optimization

### Week 5-6: Higher-Order Functions
- [ ] Enhanced standard library
- [ ] Collection processing functions
- [ ] Table-specific operations
- [ ] Utility functions

### Week 7-8: Error Handling
- [ ] Error type implementation
- [ ] Error handling patterns
- [ ] Error checking functions
- [ ] Integration with existing operations

### Week 9-10: Real-World Scenarios
- [ ] User management system
- [ ] Shopping cart system
- [ ] Data processing pipeline
- [ ] Integration testing

## Benefits of Runtime Type Checking Approach

### Simplicity
- **Minimal Implementation**: Only need to add `is` keyword and type checking logic
- **No Breaking Changes**: Existing code continues to work unchanged
- **Clear Semantics**: `x is number` is obviously a boolean expression
- **Consistent Syntax**: Works seamlessly with existing case expressions

### Functional Design
- **Boolean Results**: Type checking returns true/false, fitting functional style
- **Composable**: Can combine with logical operators (`and`, `or`, `not`)
- **Pattern Matching**: Integrates naturally with case expressions
- **No Side Effects**: Pure functions for type validation

### Extensibility
- **Easy to Add Types**: Simple to extend for new types (arrays, tuples, etc.)
- **Custom Types**: Can implement custom type checking via functions
- **Performance**: Runtime overhead only when explicitly used
- **Optional**: No requirement to use type checking in existing code

## Testing Strategy

### Unit Tests
- String operation tests
- Runtime type checking tests (`is` keyword)
- Type validation function tests
- Data structure transformation tests
- Higher-order function tests
- Error handling tests

### Integration Tests
- Real-world scenario tests
- Performance tests
- Edge case tests
- Error handling tests

### Performance Benchmarks
- Data structure operation performance
- Memory usage analysis
- Transformation efficiency
- Scalability testing

## Success Metrics

- [ ] All real-world scenarios in `tests/17_real_world_scenarios.txt` pass
- [ ] Performance within acceptable bounds (no more than 2x slower than current)
- [ ] Memory usage remains reasonable
- [ ] Code remains readable and maintainable
- [ ] Backward compatibility maintained

## Future Considerations

### Advanced Features (Post-v1.0)
- Lazy evaluation for large collections
- Parallel processing capabilities
- Advanced type system with generics
- Macro system for code generation
- Module system for code organization

### Performance Optimizations
- Structural sharing for immutable data structures
- Compile-time optimizations
- JIT compilation for hot paths
- Memory pooling for temporary objects

This plan maintains the language's functional, immutable design while adding the capabilities needed for real-world programming scenarios.