diff options
| author | elioat <{ID}+{username}@users.noreply.github.com> | 2025-08-20 14:37:26 -0400 |
|---|---|---|
| committer | elioat <{ID}+{username}@users.noreply.github.com> | 2025-08-20 14:37:26 -0400 |
| commit | ebc0c4fd8652f0c1d7f584ebec916b8452d815b9 (patch) | |
| tree | 43492cb2ccd63f51e733b003f0c189a05549cfc8 /bash | |
| parent | cc0b3832de2ae4c1518449dd32e4e87ca8a15a11 (diff) | |
| download | tour-ebc0c4fd8652f0c1d7f584ebec916b8452d815b9.tar.gz | |
*
Diffstat (limited to 'bash')
| -rw-r--r-- | bash/talk-to-computer/README.md | 359 | ||||
| -rwxr-xr-x | bash/talk-to-computer/common.sh | 234 | ||||
| -rwxr-xr-x | bash/talk-to-computer/config.sh | 126 | ||||
| -rwxr-xr-x | bash/talk-to-computer/puzzle_old | 407 | ||||
| -rw-r--r--[-rwxr-xr-x] | bash/talk-to-computer/quality_guard.sh | 29 | ||||
| -rwxr-xr-x | bash/talk-to-computer/socratic | 36 | ||||
| -rwxr-xr-x | bash/talk-to-computer/test_framework.sh | 434 |
7 files changed, 1049 insertions, 576 deletions
diff --git a/bash/talk-to-computer/README.md b/bash/talk-to-computer/README.md index 0d2603a..d702a32 100644 --- a/bash/talk-to-computer/README.md +++ b/bash/talk-to-computer/README.md @@ -1,26 +1,27 @@ -# AI Thinking Mechanisms - Intelligent AI Response Orchestration +# AI Thinking Mechanisms -A sophisticated Bash-based system that intelligently routes user prompts through specialized AI thinking strategies, leveraging multiple LLM models for comprehensive problem-solving and analysis. +A Bash-based system that routes user prompts through different AI thinking mechanisms. It uses pattern matching and basic analysis to select from multiple approaches: direct response, socratic questioning, exploration, critique, consensus, synthesis, and puzzle solving. ## Architecture Overview -The system implements a **multi-layered orchestration architecture** that combines intelligent routing, specialized thinking mechanisms, and comprehensive logging to deliver optimal AI responses. +The system processes prompts through several stages: ``` -User Prompt → Computer (Dispatcher) → Advanced Classifier → Specialized Mechanism → LLM Models → Response - ↓ ↓ ↓ ↓ ↓ - Validation Route Analysis Multi-Layer Multi-Phase Model Selection - ↓ ↓ ↓ ↓ ↓ - Sanitization Mechanism Selection Classification Processing Fallback Handling +User Prompt → Computer (Dispatcher) → Classifier → Mechanism → LLM Models → Quality Guard → Response + ↓ ↓ ↓ ↓ ↓ ↓ + Validation Pattern Analysis Selection Processing Model Calls Quality Check + ↓ ↓ ↓ ↓ ↓ ↓ + Sanitization Keyword Matching Routing Execution Fallbacks Error Handling ``` ### Core Components -- **`computer`** - Intelligent dispatcher and main entry point -- **`classifier.sh`** - Advanced multi-layer prompt classification system -- **`logging.sh`** - Unified logging, metrics, and validation system -- **Specialized Thinking Mechanisms** - Domain-specific AI response strategies -- **Model Management** - Intelligent LLM selection with fallback handling +- **`computer`** - Main dispatcher script +- **`classifier.sh`** - Pattern-based prompt classification +- **`logging.sh`** - Logging and validation utilities +- **`quality_guard.sh`** - Basic response quality monitoring +- **Thinking Mechanisms** - Different AI interaction patterns +- **Model Management** - Basic model selection and fallback handling ## Getting Started @@ -37,7 +38,7 @@ User Prompt → Computer (Dispatcher) → Advanced Classifier → Specialized Me 2. Ensure all scripts have execute permissions: ```bash chmod +x computer exploration consensus socratic critique peer-review synthesis puzzle - chmod +x logging.sh classifier.sh + chmod +x logging.sh classifier.sh quality_guard.sh ``` 3. Verify Ollama is running and accessible: ```bash @@ -60,70 +61,103 @@ User Prompt → Computer (Dispatcher) → Advanced Classifier → Specialized Me ./computer "Complex question" 3 ``` -## Advanced Prompt Classification +## Quality Guard System -### Multi-Layer Classification System +### Basic Response Quality Monitoring -The system now uses a sophisticated **three-layer classification approach**: +The Quality Guard provides simple monitoring and error handling for AI responses. -#### **Layer 1: Intent Pattern Analysis** +#### **What It Does** + +- Monitors basic response characteristics +- Detects obvious quality issues +- Provides fallback responses when needed +- Attempts to regenerate responses up to 2 times + +#### **How It Works** + +1. **Basic Checks** - Simple pattern matching for obvious issues +2. **Scoring** - Basic heuristics for response quality +3. **Retry Logic** - Up to 2 attempts to get better responses +4. **Fallbacks** - Generic helpful responses when retries fail + +#### **Limitations** + +- Uses simple pattern matching, not advanced analysis +- May not catch subtle quality issues +- Fallback responses are generic +- Not a substitute for careful prompt engineering + +#### **Configuration Options** + +```bash +# Basic threshold adjustments +export MIN_RESPONSE_LENGTH=30 # Minimum words required +export MAX_REPETITION_RATIO=0.4 # Maximum repetition allowed +export MAX_NONSENSE_SCORE=0.6 # Maximum nonsense score +export DEGRADATION_THRESHOLD=0.65 # Quality threshold for correction +export MAX_CORRECTION_ATTEMPTS=2 # Number of correction attempts +export FALLBACK_ENABLED=true # Enable fallback responses +``` + +## Prompt Classification + +### Basic Pattern Matching + +The system uses keyword and pattern matching to route prompts to different mechanisms: + +#### **Pattern Matching Rules** - **Question type detection**: what/when/where → DIRECT, why/how → SOCRATIC - **Action-oriented patterns**: improve → CRITIQUE, compare → EXPLORATION - **Puzzle & coding patterns**: algorithm/implement → PUZZLE, challenge/problem → PUZZLE - **Context-aware scoring**: strategy/planning → EXPLORATION, analysis → SOCRATIC -- **Weighted scoring system** with priority handling - -#### **Layer 2: Semantic Analysis** -- **LLM-powered classification** using natural language understanding -- **Contextual interpretation** beyond simple keyword matching -- **Confidence scoring** from 0.0-1.0 for each classification -- **Format validation** with intelligent fallback handling - -#### **Layer 3: Complexity Analysis** -- **Multi-dimensional complexity scoring**: - - Word count analysis (5 words → simple, 50+ → complex) - - Multiple questions detection - - Sentence structure analysis -- **Automatic complexity adjustments**: - - Low complexity → suggests DIRECT - - High complexity → boosts sophisticated mechanisms - -### Confidence-Based Decision Making - -- **High confidence** (>0.7): Use recommended mechanism -- **Medium confidence** (0.6-0.7): Proceed with warning -- **Low confidence** (<0.6): Default to DIRECT with user notification -- **Transparent reporting** of classification confidence and reasoning +- **Basic scoring system** with keyword matching + +#### **Basic Analysis** +- **Word count analysis**: Short prompts → DIRECT, longer → complex mechanisms +- **Keyword presence**: Simple keyword matching for routing decisions +- **Basic confidence scoring**: Simple scoring mechanism + +#### **Limitations** +- Relies on keyword matching, not deep understanding +- May misclassify prompts without obvious keywords +- Confidence scores are basic heuristics, not accurate measures +- Not a substitute for manual routing when precision matters + +### Decision Making + +- **Basic confidence scoring** from pattern matching +- **Keyword-based routing** with fallback to DIRECT +- **Simple word count analysis** for complexity estimation ### Classification Examples ```bash -# Strategic Planning (High Confidence) +# Strategic Planning Input: "What are the different approaches to solve climate change?" -Output: EXPLORATION:1.00 (correctly identifies option comparison need) +Output: EXPLORATION:1.00 (matches "different approaches" pattern) -# Improvement Request (Pattern vs Semantic Agreement) +# Improvement Request Input: "How can we improve our development workflow?" -Pattern: CRITIQUE:1.00, Semantic: EXPLORATION:0.95 -Final: CRITIQUE:1.00 (pattern wins due to "improve" keyword strength) +Output: CRITIQUE:1.00 (matches "improve" keyword) -# Complex Analysis (Complexity Boost) +# Complex Analysis Input: "Why do you think this approach might fail and what are the underlying assumptions?" -Output: SOCRATIC:0.85 (correctly identifies deep questioning need) +Output: SOCRATIC:0.85 (matches "why" and complexity indicators) -# Simple Question (Complexity Override) +# Simple Question Input: "What is 2+2?" -Output: DIRECT:0.8 (low complexity override) +Output: DIRECT:0.8 (simple, short question) -# Algorithm Challenge (High Confidence) +# Algorithm Challenge Input: "How can I implement a binary search algorithm?" -Output: PUZZLE:1.00 (correctly identifies coding challenge need) +Output: PUZZLE:1.00 (matches "algorithm" and "implement" keywords) ``` ## Thinking Mechanisms -### 1. **Exploration** - Systematic Path Analysis -**Best for**: Strategic planning, comparing alternatives, exploring options +### 1. **Exploration** - Multiple Path Analysis +**Purpose**: Generate multiple solution approaches and compare them ```bash ./exploration -p 4 "How can we improve our development process?" @@ -131,37 +165,43 @@ Output: PUZZLE:1.00 (correctly identifies coding challenge need) **Process**: - **Phase 1**: Generate multiple solution paths -- **Phase 2**: Deep-dive analysis of each path -- **Phase 3**: Comparative analysis and recommendations +- **Phase 2**: Basic analysis of each path +- **Phase 3**: Simple comparison and recommendations -### 2. **Consensus** - Multi-Model Agreement -**Best for**: Critical decisions, reducing bias, improving reliability +**Notes**: Uses multiple LLM calls to generate different approaches + +### 2. **Consensus** - Multiple Model Responses +**Purpose**: Get responses from multiple models and compare them ```bash ./consensus "What's the best approach to this problem?" ``` **Process**: -- **Phase 1**: Independent model responses -- **Phase 2**: Confidence validation -- **Phase 3**: Cross-model voting -- **Phase 4**: Consensus determination +- **Phase 1**: Get responses from multiple models +- **Phase 2**: Basic comparison +- **Phase 3**: Simple voting mechanism +- **Phase 4**: Combine responses + +**Notes**: Limited by available models and simple comparison logic -### 3. **Socratic** - Deep Questioning -**Best for**: Complex analysis, uncovering assumptions, deep understanding +### 3. **Socratic** - Question-Based Analysis +**Purpose**: Use AI-generated questions to analyze prompts ```bash ./socratic "Explain the implications of this decision" ``` **Process**: -- **Phase 1**: Initial response generation -- **Phase 2**: Critical questioning -- **Phase 3**: Deep analysis -- **Phase 4**: Synthesis and conclusions +- **Phase 1**: Generate initial response +- **Phase 2**: Generate follow-up questions +- **Phase 3**: Get responses to questions +- **Phase 4**: Combine into final output -### 4. **Critique** - Improvement & Refinement -**Best for**: Code review, document improvement, quality enhancement +**Notes**: Creates a back-and-forth conversation between AI models + +### 4. **Critique** - Improvement Analysis +**Purpose**: Get improvement suggestions for code or text ```bash ./critique -f code.py "How can we improve this code?" @@ -169,36 +209,42 @@ Output: PUZZLE:1.00 (correctly identifies coding challenge need) **Process**: - **Phase 1**: Initial assessment -- **Phase 2**: Detailed critique -- **Phase 3**: Improvement suggestions -- **Phase 4**: Implementation guidance +- **Phase 2**: Generate critique +- **Phase 3**: Suggest improvements +- **Phase 4**: Provide guidance + +**Notes**: Basic improvement suggestions based on AI analysis -### 5. **Peer Review** - Collaborative Feedback -**Best for**: Team collaboration, multiple perspectives, comprehensive review +### 5. **Peer Review** - Multiple AI Reviewers +**Purpose**: Get feedback from multiple AI perspectives ```bash ./peer-review "Review this proposal" ``` **Process**: -- **Phase 1**: Multiple reviewer perspectives -- **Phase 2**: Feedback consolidation -- **Phase 3**: Actionable recommendations +- **Phase 1**: Generate multiple reviews +- **Phase 2**: Basic consolidation +- **Phase 3**: Combine feedback + +**Notes**: Simple multiple AI review approach -### 6. **Synthesis** - Integration & Unification -**Best for**: Combining approaches, integrating solutions, creating unified strategies +### 6. **Synthesis** - Combine Approaches +**Purpose**: Combine multiple approaches into one ```bash ./synthesis "How can we combine these different approaches?" ``` **Process**: -- **Phase 1**: Approach identification -- **Phase 2**: Integration analysis -- **Phase 3**: Unified solution creation +- **Phase 1**: Identify approaches +- **Phase 2**: Basic analysis +- **Phase 3**: Simple combination -### 7. **Puzzle** - Problem Solving & Coding -**Best for**: Algorithm challenges, coding problems, puzzle solving, programming education +**Notes**: Basic approach combination mechanism + +### 7. **Puzzle** - Coding Problem Solving +**Purpose**: Help with coding problems and algorithms ```bash ./puzzle "How can I implement a sorting algorithm?" @@ -206,30 +252,32 @@ Output: PUZZLE:1.00 (correctly identifies coding challenge need) ``` **Process**: -- **Phase 1**: Problem analysis and classification -- **Phase 2**: Solution strategy development -- **Phase 3**: Implementation guidance with code examples -- **Phase 4**: Solution validation and testing - -**Special Features**: -- **Built-in Lil language expertise** for RAG-like context -- **Algorithmic complexity analysis** and optimization tips -- **Multiple solution approaches** with trade-off analysis -- **Language-specific code examples** and best practices -- **Testing and debugging guidance** for robust solutions +- **Phase 1**: Basic problem analysis +- **Phase 2**: Solution approach +- **Phase 3**: Code examples +- **Phase 4**: Basic validation +- **Phase 5**: Optional Lil code testing if available + +**Features**: +- Basic Lil language knowledge +- Simple algorithm analysis +- Code examples in different languages +- Basic testing guidance + +**Notes**: Limited by available Lil testing tools ## Configuration ### Model Selection -The system automatically selects appropriate models for different tasks: +Configure which models to use for different tasks: ```bash -# Exploration models +# Models for different mechanisms EXPLORATION_MODEL="llama3:8b-instruct-q4_K_M" ANALYSIS_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" -# Consensus models (multiple) +# Models for consensus mechanism MODELS=( "llama3:8b-instruct-q4_K_M" "phi3:3.8b-mini-4k-instruct-q4_K_M" @@ -238,17 +286,16 @@ MODELS=( "dolphin3:latest" ) -# Classification model +# Model for classification CLASSIFIER_MODEL="gemma3n:e2b" ``` -### Automatic Fallbacks +### Basic Fallbacks -The system includes intelligent fallback mechanisms: -- **Model validation** before execution -- **Automatic fallback** to available models -- **Graceful degradation** when preferred models unavailable -- **Classification fallbacks** when semantic analysis fails +The system has basic fallback mechanisms: +- Model availability checking +- Fallback to alternative models when primary models unavailable +- Basic error handling for model failures ## Logging & Metrics @@ -261,6 +308,7 @@ All sessions are logged with comprehensive metadata: - **Model selection** decisions - **Full conversation** transcripts - **Error handling** details +- **Quality monitoring** results and correction attempts ### Performance Metrics @@ -283,6 +331,8 @@ get_metrics_summary # Classification logs ~/tmp/ai_thinking/classification.log + +# Quality monitoring logs (integrated into session files) ``` ## Security & Validation @@ -301,6 +351,7 @@ get_metrics_summary - **User-friendly** error messages with actionable guidance - **Fallback mechanisms** for critical failures - **Input validation** with clear error reporting +- **Quality degradation** protection with automatic correction ## Advanced Features @@ -337,6 +388,15 @@ The `computer` script automatically routes prompts based on: - **Context-aware** mechanism selection - **Optimal mechanism** selection with fallbacks +### Quality Monitoring Integration + +All thinking mechanisms now include: +- **Automatic quality assessment** of every LLM response +- **Degradation detection** with pattern recognition +- **Automatic correction** attempts for poor quality outputs +- **Intelligent fallbacks** when correction fails +- **Mechanism-specific** quality relevance checking + ## Testing & Validation ### Validation Functions @@ -354,6 +414,11 @@ validate_model "model_name" "fallback_model" # Classification testing classify_prompt "Test prompt" false # Pattern-only mode classify_prompt "Test prompt" true # Full semantic mode + +# Quality monitoring testing +assess_quality "response" "context" "mechanism" +detect_degradation_patterns "response" +guard_output_quality "response" "context" "mechanism" "model" ``` ### Error Testing @@ -364,6 +429,9 @@ classify_prompt "Test prompt" true # Full semantic mode ./computer -f nonexistent.txt # Missing file ./computer -x # Invalid option ./computer "test" 10 # Invalid rounds + +# Test quality monitoring +./test_quality_guard.sh # Quality guard system test ``` ## Performance Considerations @@ -375,6 +443,7 @@ classify_prompt "Test prompt" true # Full semantic mode - **Minimal overhead** for simple queries - **Classification caching** for repeated patterns - **Parallel processing** where applicable +- **Quality monitoring** with minimal performance impact ### Resource Management @@ -383,6 +452,7 @@ classify_prompt "Test prompt" true # Full semantic mode - **Memory-efficient** processing - **Graceful timeout** handling - **Classification result** caching +- **Quality assessment** optimization ## Troubleshooting @@ -411,6 +481,12 @@ classify_prompt "Test prompt" true # Full semantic mode - Consider using -d flag for direct responses - Review prompt clarity and specificity +6. **"Quality below threshold"** + - System automatically attempts correction + - Check quality monitoring logs for details + - Fallback responses ensure helpful output + - Consider rephrasing complex prompts + ### Debug Mode ```bash @@ -423,29 +499,31 @@ tail -f ~/tmp/ai_thinking/errors.log # Test classification directly source classifier.sh classify_prompt "Your test prompt" true + +# Test quality monitoring +source quality_guard.sh +assess_quality "test response" "test context" "puzzle" ``` -## Future Enhancements +## Future Improvements -### Planned Features +### Possible Enhancements -- **Configuration file** support for model preferences -- **Plugin system** for custom thinking mechanisms -- **API integration** for external LLM services -- **Advanced caching** for repeated queries -- **Progress indicators** for long-running operations -- **Unit test** framework for validation -- **Classification learning** from user feedback +- Additional thinking mechanisms +- More model integration options +- Improved caching +- Better error handling +- Enhanced testing +- Performance optimizations ### Extensibility -The modular architecture makes it easy to: -- **Add new thinking mechanisms** -- **Integrate additional models** -- **Customize validation rules** -- **Extend logging capabilities** -- **Add new classification patterns** -- **Implement custom confidence algorithms** +The basic modular structure allows for: +- Adding new mechanisms +- Integrating additional models +- Modifying validation rules +- Extending logging +- Updating classification patterns ## Examples @@ -488,6 +566,15 @@ The modular architecture makes it easy to: ./computer "Compare different approaches to implementing microservices" ``` +### Quality Monitoring in Action + +```bash +# All mechanisms now include quality protection +./computer "Complex question requiring deep analysis" 3 +# Quality monitoring automatically protects against degradation +# Fallback responses ensure helpful output even with poor LLM performance +``` + ## Contributing ### Development Guidelines @@ -498,29 +585,25 @@ The modular architecture makes it easy to: - **Include validation** for all inputs and parameters - **Test thoroughly** with various input types and edge cases - **Update classification** patterns when adding new mechanisms +- **Integrate quality monitoring** for all new LLM interactions +- **Follow quality guard** patterns for consistent protection ### Code Style -- **Consistent naming** conventions -- **Clear comments** explaining complex logic -- **Error handling** for all external calls -- **Modular functions** with single responsibilities -- **Validation functions** for all inputs -- **Comprehensive logging** for debugging +- Consistent naming conventions +- Clear comments explaining complex logic +- Error handling for all external calls +- Modular functions with single responsibilities +- Validation functions for all inputs +- Comprehensive logging for debugging +- Quality monitoring integration for all AI responses ## License This project is open source. Please refer to individual script headers for specific licensing information. -## Support - -For issues, questions, or contributions: -1. Check the troubleshooting section above -2. Review error logs in `~/tmp/ai_thinking/` -3. Verify system requirements and dependencies -4. Test with simple examples first -5. Check classification logs for routing decisions - --- -**Note**: This system is designed for intelligent AI response orchestration. The quality of responses depends on the available LLM models and the complexity of the input prompts. Always review and validate AI-generated content for critical applications.
\ No newline at end of file +**Note**: This is a basic system for routing prompts to different AI thinking approaches. Response quality depends on the underlying LLM models and prompt quality. Manual review is recommended for important applications. + +The Quality Guard provides basic response monitoring but is not a substitute for careful prompt engineering or critical review of AI outputs. diff --git a/bash/talk-to-computer/common.sh b/bash/talk-to-computer/common.sh new file mode 100755 index 0000000..4f11ffe --- /dev/null +++ b/bash/talk-to-computer/common.sh @@ -0,0 +1,234 @@ +#!/bin/bash + +# Common functionality shared across all AI thinking mechanisms +# This file contains utilities and initialization code used by multiple scripts + +# --- Script Directory Setup --- +# Get the directory where this script is located +get_script_dir() { + echo "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +} + +# --- Initialization Functions --- + +# Initialize a thinking mechanism script with common dependencies +init_thinking_mechanism() { + local script_path="$1" + + # Set up script directory + SCRIPT_DIR="$(cd "$(dirname "${script_path}")" && pwd)" + + # Source configuration + source "${SCRIPT_DIR}/config.sh" + + # Source logging system + source "${SCRIPT_DIR}/logging.sh" + + # Source quality guard for output quality protection + source "${SCRIPT_DIR}/quality_guard.sh" + + # Get mechanism name automatically + MECHANISM_NAME=$(get_mechanism_name "$script_path") + + # Set up resource management + setup_cleanup_trap + + export SCRIPT_DIR MECHANISM_NAME +} + +# Initialize the main dispatcher with common dependencies +init_dispatcher() { + local script_path="$1" + + # Set up script directory + SCRIPT_DIR="$(cd "$(dirname "${script_path}")" && pwd)" + + # Source logging system (dispatcher sources this later) + # source "${SCRIPT_DIR}/logging.sh" + + export SCRIPT_DIR +} + +# --- Model Validation Functions --- + +# Validate and set a model with fallback +validate_and_set_model() { + local model_var="$1" + local model_name="$2" + local fallback_model="$3" + + local validated_model + validated_model=$(validate_model "$model_name" "$fallback_model") + + if [ $? -ne 0 ]; then + log_error "No valid model available for $model_var" + return 1 + fi + + eval "$model_var=\"$validated_model\"" + echo "Set $model_var to: $validated_model" +} + +# --- Argument Processing Functions --- + +# Common file path validation +validate_file_arg() { + local file_path="$1" + + if [ -n "$file_path" ]; then + validate_file_path "$file_path" + if [ $? -ne 0 ]; then + return 1 + fi + fi + + echo "$file_path" +} + +# --- Cleanup Functions --- + +# Global array to track resources for cleanup +declare -a CLEANUP_RESOURCES=() + +# Register a resource for cleanup +register_cleanup_resource() { + local resource="$1" + CLEANUP_RESOURCES+=("$resource") +} + +# Clean up temporary resources +cleanup_resources() { + local exit_code=$? + + # Clean up registered resources + for resource in "${CLEANUP_RESOURCES[@]}"; do + if [ -d "$resource" ]; then + rm -rf "$resource" 2>/dev/null || true + elif [ -f "$resource" ]; then + rm -f "$resource" 2>/dev/null || true + fi + done + + # Clean up any additional temp directories + if [ -n "$TEMP_DIR" ] && [ -d "$TEMP_DIR" ]; then + rm -rf "$TEMP_DIR" 2>/dev/null || true + fi + + exit $exit_code +} + +# Set up trap for cleanup on script exit +setup_cleanup_trap() { + trap cleanup_resources EXIT INT TERM +} + +# Create a temporary directory with automatic cleanup +create_managed_temp_dir() { + local prefix="${1:-ai_thinking}" + local temp_dir + + temp_dir=$(mktemp -d -t "${prefix}_XXXXXX") + register_cleanup_resource "$temp_dir" + + echo "$temp_dir" +} + +# Create a temporary file with automatic cleanup +create_managed_temp_file() { + local prefix="${1:-ai_thinking}" + local suffix="${2:-tmp}" + local temp_file + + temp_file=$(mktemp -t "${prefix}_XXXXXX.${suffix}") + register_cleanup_resource "$temp_file" + + echo "$temp_file" +} + +# --- Standardized Error Handling --- + +# Standardized error codes +ERROR_INVALID_ARGUMENT=1 +ERROR_FILE_NOT_FOUND=2 +ERROR_MODEL_UNAVAILABLE=3 +ERROR_VALIDATION_FAILED=4 +ERROR_PROCESSING_FAILED=5 +ERROR_RESOURCE_ERROR=6 + +# Standardized error handling function +handle_error() { + local error_code="$1" + local error_message="$2" + local script_name="${3:-$(basename "${BASH_SOURCE[1]}")}" + local line_number="${4:-${BASH_LINENO[0]}}" + + # Log the error + log_error "[$script_name:$line_number] $error_message" + + # Print user-friendly error message + echo "Error: $error_message" >&2 + + # Exit with appropriate code + exit "$error_code" +} + +# Validation error handler +handle_validation_error() { + local error_message="$1" + local script_name="${2:-$(basename "${BASH_SOURCE[1]}")}" + + handle_error "$ERROR_VALIDATION_FAILED" "$error_message" "$script_name" "${BASH_LINENO[0]}" +} + +# Model error handler +handle_model_error() { + local model_name="$1" + local script_name="${2:-$(basename "${BASH_SOURCE[1]}")}" + + handle_error "$ERROR_MODEL_UNAVAILABLE" "Model '$model_name' is not available" "$script_name" "${BASH_LINENO[0]}" +} + +# File error handler +handle_file_error() { + local file_path="$1" + local operation="$2" + local script_name="${3:-$(basename "${BASH_SOURCE[1]}")}" + + handle_error "$ERROR_FILE_NOT_FOUND" "Cannot $operation file: $file_path" "$script_name" "${BASH_LINENO[0]}" +} + +# Processing error handler +handle_processing_error() { + local operation="$1" + local details="$2" + local script_name="${3:-$(basename "${BASH_SOURCE[1]}")}" + + handle_error "$ERROR_PROCESSING_FAILED" "Failed to $operation: $details" "$script_name" "${BASH_LINENO[0]}" +} + +# --- Utility Functions --- + +# Check if a command exists +command_exists() { + command -v "$1" >/dev/null 2>&1 +} + +# Create a temporary directory with cleanup +create_temp_dir() { + local prefix="${1:-ai_thinking}" + local temp_dir + + temp_dir=$(mktemp -d -t "${prefix}_XXXXXX") + echo "$temp_dir" +} + +# --- Common Constants --- + +# Default values +DEFAULT_ROUNDS=2 +DEFAULT_MODEL="gemma3n:e2b" + +# File paths +LOG_DIR=~/tmp/ai_thinking + +export DEFAULT_ROUNDS DEFAULT_MODEL LOG_DIR diff --git a/bash/talk-to-computer/config.sh b/bash/talk-to-computer/config.sh new file mode 100755 index 0000000..ec612cc --- /dev/null +++ b/bash/talk-to-computer/config.sh @@ -0,0 +1,126 @@ +#!/bin/bash + +# Centralized Configuration File +# This file contains all model configurations, defaults, and system settings +# for the AI thinking mechanisms system. + +# --- Default Models --- + +# Main dispatcher model +DEFAULT_MODEL="gemma3n:e2b" + +# Classification model +CLASSIFIER_MODEL="gemma3n:e2b" + +# --- Thinking Mechanism Models --- + +# Exploration mechanism +EXPLORATION_MODEL="llama3:8b-instruct-q4_K_M" +ANALYSIS_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" + +# Consensus mechanism +CONSENSUS_MODELS=( + "llama3:8b-instruct-q4_K_M" + "phi3:3.8b-mini-4k-instruct-q4_K_M" + "deepseek-r1:1.5b" + "gemma3n:e2b" + "dolphin3:latest" +) +CONSENSUS_JUDGE_MODEL="gemma3n:e2b" + +# Socratic mechanism +SOCRATIC_RESPONSE_MODEL="llama3:8b-instruct-q4_K_M" +SOCRATIC_QUESTION_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" + +# Critique mechanism +CRITIQUE_MODEL="llama3:8b-instruct-q4_K_M" + +# Synthesis mechanism +SYNTHESIS_MODEL="llama3:8b-instruct-q4_K_M" + +# Peer Review mechanism +PEER_REVIEW_MODEL="llama3:8b-instruct-q4_K_M" + +# Puzzle mechanism +PUZZLE_MODEL="llama3:8b-instruct-q4_K_M" +PUZZLE_ANALYSIS_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" + +# --- System Settings --- + +# Default values +DEFAULT_ROUNDS=2 +DEFAULT_LANGUAGE="lil" + +# Quality Guard settings +MIN_RESPONSE_LENGTH=30 +MAX_REPETITION_RATIO=0.4 +MAX_NONSENSE_SCORE=0.6 +DEGRADATION_THRESHOLD=0.65 +MAX_CORRECTION_ATTEMPTS=2 +FALLBACK_ENABLED=true + +# Logging settings +LOG_DIR=~/tmp/ai_thinking +SESSION_LOG="${LOG_DIR}/session_$(date +%Y%m%d_%H%M%S).json" +ERROR_LOG="${LOG_DIR}/errors.log" +METRICS_FILE="${LOG_DIR}/performance_metrics.json" +CLASSIFICATION_LOG="${LOG_DIR}/classification.log" + +# Security settings +MAX_PROMPT_LENGTH=10000 + +# --- Model Fallbacks --- + +# Fallback model for any model that fails validation +FALLBACK_MODEL="gemma3n:e2b" + +# --- Environment Variable Support --- + +# Allow overriding models via environment variables +if [ -n "$AI_DEFAULT_MODEL" ]; then + DEFAULT_MODEL="$AI_DEFAULT_MODEL" +fi + +if [ -n "$AI_CLASSIFIER_MODEL" ]; then + CLASSIFIER_MODEL="$AI_CLASSIFIER_MODEL" +fi + +if [ -n "$AI_EXPLORATION_MODEL" ]; then + EXPLORATION_MODEL="$AI_EXPLORATION_MODEL" +fi + +if [ -n "$AI_ANALYSIS_MODEL" ]; then + ANALYSIS_MODEL="$AI_ANALYSIS_MODEL" +fi + +if [ -n "$AI_PUZZLE_MODEL" ]; then + PUZZLE_MODEL="$AI_PUZZLE_MODEL" +fi + +# --- Utility Functions --- + +# Get a model with fallback support +get_model_with_fallback() { + local primary_model="$1" + local fallback_model="$2" + + if [ -n "$primary_model" ]; then + echo "$primary_model" + else + echo "$fallback_model" + fi +} + +# Validate if a model is available +is_model_available() { + local model="$1" + ollama list 2>/dev/null | grep -q "$model" +} + +# Export all configuration variables +export DEFAULT_MODEL CLASSIFIER_MODEL EXPLORATION_MODEL ANALYSIS_MODEL +export CONSENSUS_MODELS CONSENSUS_JUDGE_MODEL SOCRATIC_RESPONSE_MODEL SOCRATIC_QUESTION_MODEL +export CRITIQUE_MODEL SYNTHESIS_MODEL PEER_REVIEW_MODEL PUZZLE_MODEL PUZZLE_ANALYSIS_MODEL +export DEFAULT_ROUNDS DEFAULT_LANGUAGE MIN_RESPONSE_LENGTH MAX_REPETITION_RATIO +export MAX_NONSENSE_SCORE DEGRADATION_THRESHOLD MAX_CORRECTION_ATTEMPTS FALLBACK_ENABLED +export LOG_DIR SESSION_LOG ERROR_LOG METRICS_FILE CLASSIFICATION_LOG MAX_PROMPT_LENGTH FALLBACK_MODEL diff --git a/bash/talk-to-computer/puzzle_old b/bash/talk-to-computer/puzzle_old deleted file mode 100755 index 080db80..0000000 --- a/bash/talk-to-computer/puzzle_old +++ /dev/null @@ -1,407 +0,0 @@ -#!/bin/bash - -# Get the directory where this script is located -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -# Puzzle System -# This script specializes in puzzle solving and coding challenges with Lil programming language expertise. -# -# APPLICATION LOGIC: -# The puzzle process implements a systematic problem-solving approach that combines -# algorithmic thinking, code analysis, and domain-specific knowledge. The system operates -# through four distinct phases designed to maximize puzzle-solving effectiveness: -# -# PHASE 1 - PROBLEM ANALYSIS: -# - Analyzes the puzzle or coding challenge -# - Identifies problem type and complexity -# - Determines appropriate solution strategies -# - Provides Lil language context when relevant -# -# PHASE 2 - SOLUTION STRATEGY: -# - Develops multiple solution approaches -# - Considers algorithmic complexity and trade-offs -# - Provides Lil-specific implementation guidance -# - Suggests testing and validation strategies -# -# PHASE 3 - IMPLEMENTATION GUIDANCE: -# - Provides detailed code examples in Lil -# - Explains key concepts and patterns -# - Suggests optimizations and improvements -# - Offers debugging and troubleshooting tips -# -# PHASE 4 - SOLUTION VALIDATION: -# - Reviews the proposed solution -# - Identifies potential edge cases -# - Suggests testing scenarios -# - Provides alternative approaches if applicable -# -# PUZZLE MODELING: -# The system applies systematic problem-solving principles to coding challenges: -# - Algorithmic thinking ensures optimal solutions -# - Lil language expertise provides domain-specific guidance -# - Multiple solution approaches reveal different perspectives -# - Systematic validation reduces errors and improves code quality -# -# The puzzle process emphasizes systematic analysis and practical implementation, -# ensuring users understand both the problem-solving approach and the Lil language specifics. - -# Source the logging system using absolute path -source "${SCRIPT_DIR}/logging.sh" - -# Source the Lil tester for secure code testing -source "${SCRIPT_DIR}/lil_tester.sh" - -# --- Lil Code Extraction Function --- -extract_lil_code() { - local text="$1" - local code="" - - # Look for function definitions - if echo "$text" | grep -q "on [a-zA-Z_][a-zA-Z0-9_]* do"; then - code=$(echo "$text" | grep -A 20 "on [a-zA-Z_][a-zA-Z0-9_]* do" | head -20) - elif echo "$text" | grep -q "```lil"; then - # Extract content between ```lil and ``` - code=$(echo "$text" | sed -n '/```lil/,/```/p' | sed '1d;$d') - elif echo "$text" | grep -q "```"; then - # Extract content between any ``` blocks - code=$(echo "$text" | sed -n '/```/,/```/p' | sed '1d;$d') - fi - - # Clean up the extracted code - if [ -n "$code" ]; then - # Remove markdown formatting and extra whitespace - code=$(echo "$code" | sed 's/^[[:space:]]*//' | sed 's/[[:space:]]*$//') - echo "$code" - fi -} - -# --- Model Configuration --- -PUZZLE_MODEL="llama3:8b-instruct-q4_K_M" -ANALYSIS_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" - -# Validate and set models with fallbacks -PUZZLE_MODEL=$(validate_model "$PUZZLE_MODEL" "gemma3n:e2b") -if [ $? -ne 0 ]; then - log_error "No valid puzzle model available" - exit 1 -fi - -ANALYSIS_MODEL=$(validate_model "$ANALYSIS_MODEL" "gemma3n:e2b") -if [ $? -ne 0 ]; then - log_error "No valid analysis model available" - exit 1 -fi - -# --- Defaults --- -DEFAULT_ROUNDS=2 - -# --- Argument Validation --- -if [ "$#" -lt 1 ]; then - echo -e "\n\tPuzzle" - echo -e "\tThis script specializes in puzzle solving and coding challenges with Lil programming language expertise." - echo -e "\n\tUsage: $0 [-f <file_path>] [-l <language>] \"<your puzzle/challenge>\" [number_of_rounds]" - echo -e "\n\tExample: $0 -f ./challenge.txt -l lil \"How can I implement a sorting algorithm?\" 2" - echo -e "\n\tIf number_of_rounds is not provided, the program will default to 2 rounds." - echo -e "\n\t-f <file_path> (optional): Append the contents of the file to the prompt." - echo -e "\n\t-l <language> (optional): Specify programming language focus (default: lil)." - echo -e "\n" - exit 1 -fi - -# --- Argument Parsing --- -FILE_PATH="" -LANGUAGE="lil" -while getopts "f:l:" opt; do - case $opt in - f) - FILE_PATH="$OPTARG" - ;; - l) - LANGUAGE="$OPTARG" - ;; - *) - log_error "Invalid option: -$OPTARG" - exit 1 - ;; - esac -done -shift $((OPTIND -1)) - -PROMPT="$1" -if [ -z "$2" ]; then - ROUNDS=2 -else - ROUNDS=$2 -fi - -# Validate prompt -PROMPT=$(validate_prompt "$PROMPT") -if [ $? -ne 0 ]; then - exit 1 -fi - -# Validate file path if provided -if [ -n "$FILE_PATH" ]; then - if ! validate_file_path "$FILE_PATH"; then - exit 1 - fi - FILE_CONTENTS=$(cat "$FILE_PATH") - PROMPT="$PROMPT\n[FILE CONTENTS]\n$FILE_CONTENTS\n[END FILE]" -fi - -# Validate rounds -if ! [[ "$ROUNDS" =~ ^[1-9][0-9]*$ ]] || [ "$ROUNDS" -gt 5 ]; then - log_error "Invalid number of rounds: $ROUNDS (must be 1-5)" - exit 1 -fi - -# --- File Initialization --- -mkdir -p ~/tmp -SESSION_FILE=~/tmp/puzzle_$(date +%Y%m%d_%H%M%S).txt - -# Initialize timing -SESSION_ID=$(generate_session_id) -start_timer "$SESSION_ID" "puzzle" - -echo "Puzzle Session Log: ${SESSION_FILE}" -echo "---------------------------------" - -# Store the initial user prompt in the session file -echo "USER PROMPT: ${PROMPT}" >> "${SESSION_FILE}" -echo "LANGUAGE FOCUS: ${LANGUAGE}" >> "${SESSION_FILE}" -echo "NUMBER OF ROUNDS: ${ROUNDS}" >> "${SESSION_FILE}" -echo "" >> "${SESSION_FILE}" - -# --- Lil Language Knowledge Base --- -get_lil_knowledge() { - cat << 'EOF' -LIL PROGRAMMING LANGUAGE KNOWLEDGE BASE: - -Lil is a scripting language designed for Decker, a multimedia creative tool. Key features: - -DATA TYPES: -- Numbers: 42, 37.5, -29999 (floating-point) -- Strings: apple, foo bar (with escape sequences) -- Lists: (), (1,2,3), list 42 -- Dictionaries: dict, key-value pairs -- Tables: rectangular arrays with string keys -- Functions: declared with 'on' -- nil: represents absence of value - -CORE SYNTAX: -- Assignment: x:42 -- Function calls: function[arg1 arg2] -- Comments: # line comment -- Function definition: on name do ... end -- Conditionals: if condition ... else ... end -- Loops: each x in list do ... end, while condition do ... end - -OPERATORS: -- Arithmetic: +, -, *, /, %, ^ -- Comparison: =, <, > (no !=, <=, >=) -- Logical: & (AND), | (OR), ! (NOT) -- List operations: , (join), take, drop -- String operations: fuse (concatenate), split -- Table operations: join, cross - -QUERY SYNTAX: -- select/extract/update with where/by/orderby clauses -- Example: first extract key orderby value desc from dict - -COMMON PATTERNS: -- Dictionary iteration: each k in keys dict do ... end -- List processing: each x in list do ... end -- Table manipulation: rows table, cols table -- Error handling: x unless y (nil-coalescing) -- Set operations using dictionaries - -UNIQUE FEATURES: -- Right-to-left operator precedence -- Soft type system with automatic conversions -- Query language for data manipulation -- Vector-oriented operations -- Built-in table operations -EOF -} - -# --- Phase 1: Problem Analysis --- -echo "Phase 1: Analyzing the puzzle or coding challenge..." -echo "PHASE 1 - PROBLEM ANALYSIS:" >> "${SESSION_FILE}" - -PROBLEM_ANALYSIS_PROMPT="You are an expert puzzle solver and programming mentor specializing in the ${LANGUAGE} programming language. Analyze the following puzzle or coding challenge. - -CHALLENGE: ${PROMPT} - -Please provide a comprehensive analysis including: -1. Problem type classification (algorithm, data structure, logic, etc.) -2. Complexity assessment (time/space requirements) -3. Key concepts and patterns involved -4. Relevant ${LANGUAGE} language features that could help -5. Potential solution approaches -6. Common pitfalls or edge cases to consider - -Provide a clear, structured analysis that helps understand the problem." - -problem_analysis=$(ollama run "${PUZZLE_MODEL}" "${PROBLEM_ANALYSIS_PROMPT}") - -echo "PROBLEM ANALYSIS:" >> "${SESSION_FILE}" -echo "${problem_analysis}" >> "${SESSION_FILE}" -echo "" >> "${SESSION_FILE}" - -# --- Phase 2: Solution Strategy --- -echo "Phase 2: Developing solution strategies..." -echo "PHASE 2 - SOLUTION STRATEGY:" >> "${SESSION_FILE}" - -SOLUTION_STRATEGY_PROMPT="Based on the problem analysis, develop multiple solution strategies for this puzzle or coding challenge. - -ORIGINAL CHALLENGE: ${PROMPT} - -PROBLEM ANALYSIS: ${problem_analysis} - -Please provide: -1. At least 2-3 different solution approaches -2. Algorithmic complexity analysis for each approach -3. Trade-offs between different solutions -4. Specific ${LANGUAGE} language constructs that would be useful -5. Implementation considerations and challenges -6. Testing and validation strategies - -Focus on practical, implementable solutions with clear reasoning." - -solution_strategy=$(ollama run "${PUZZLE_MODEL}" "${SOLUTION_STRATEGY_PROMPT}") - -echo "SOLUTION STRATEGY:" >> "${SESSION_FILE}" -echo "${solution_strategy}" >> "${SESSION_FILE}" -echo "" >> "${SESSION_FILE}" - -# --- Phase 3: Implementation Guidance --- -echo "Phase 3: Providing implementation guidance..." -echo "PHASE 3 - IMPLEMENTATION GUIDANCE:" >> "${SESSION_FILE}" - -IMPLEMENTATION_PROMPT="Provide detailed implementation guidance for the best solution approach to this puzzle or coding challenge. - -ORIGINAL CHALLENGE: ${PROMPT} -PROBLEM ANALYSIS: ${problem_analysis} -SOLUTION STRATEGY: ${solution_strategy} - -Please provide: -1. Step-by-step implementation plan -2. Complete code example in ${LANGUAGE} (if applicable) -3. Explanation of key code sections and patterns -4. Variable naming and structure recommendations -5. Error handling and edge case considerations -6. Performance optimization tips -7. Testing and debugging guidance - -Make the implementation clear and educational, explaining the reasoning behind each decision." - -implementation_guidance=$(ollama run "${PUZZLE_MODEL}" "${IMPLEMENTATION_PROMPT}") - -echo "IMPLEMENTATION GUIDANCE:" >> "${SESSION_FILE}" -echo "${implementation_guidance}" >> "${SESSION_FILE}" -echo "" >> "${SESSION_FILE}" - -# --- Phase 4: Code Testing (if applicable) --- -if [[ "$LANGUAGE" == "lil" ]] && [[ "$implementation_guidance" =~ (on [a-zA-Z_][a-zA-Z0-9_]* do|function|procedure) ]]; then - echo "Phase 4: Testing the Lil code implementation..." - echo "PHASE 4 - CODE TESTING:" >> "${SESSION_FILE}" - - # Extract Lil code from implementation guidance - lil_code=$(extract_lil_code "$implementation_guidance") - - if [ -n "$lil_code" ]; then - echo "Extracted Lil code for testing:" - echo "----------------------------------------" - echo "$lil_code" - echo "----------------------------------------" - - # Test the extracted code - test_result=$(test_lil_script "$lil_code" "puzzle_implementation") - test_exit_code=$? - - echo "CODE TESTING RESULTS:" >> "${SESSION_FILE}" - echo "$test_result" >> "${SESSION_FILE}" - echo "" >> "${SESSION_FILE}" - - if [ $test_exit_code -eq 0 ]; then - echo "✅ Lil code testing PASSED" - else - echo "❌ Lil code testing FAILED" - echo "Note: The code may have syntax errors or runtime issues." - fi - else - echo "No executable Lil code found in implementation guidance." - echo "CODE TESTING: No executable code found" >> "${SESSION_FILE}" - fi -else - echo "Phase 4: Skipping code testing (not Lil language or no executable code)" - echo "CODE TESTING: Skipped (not applicable)" >> "${SESSION_FILE}" -fi - -# --- Phase 5: Solution Validation --- -echo "Phase 5: Validating and reviewing the solution..." -echo "PHASE 5 - SOLUTION VALIDATION:" >> "${SESSION_FILE}" - -VALIDATION_PROMPT="Review and validate the proposed solution to ensure it's correct, efficient, and well-implemented. - -ORIGINAL CHALLENGE: ${PROMPT} -PROBLEM ANALYSIS: ${problem_analysis} -SOLUTION STRATEGY: ${solution_strategy} -IMPLEMENTATION: ${implementation_guidance} - -Please provide: -1. Code review and correctness verification -2. Edge case analysis and testing scenarios -3. Performance analysis and optimization opportunities -4. Alternative approaches or improvements -5. Common mistakes to avoid -6. Learning resources and next steps -7. Final recommendations and best practices - -Ensure the solution is robust, maintainable, and follows ${LANGUAGE} best practices." - -solution_validation=$(ollama run "${ANALYSIS_MODEL}" "${VALIDATION_PROMPT}") - -echo "SOLUTION VALIDATION:" >> "${SESSION_FILE}" -echo "${solution_validation}" >> "${SESSION_FILE}" - -# End timing -duration=$(end_timer "$SESSION_ID" "puzzle") - -# --- Final Output --- -echo "---------------------------------" -echo "Puzzle-solving process complete." -echo "---------------------------------" -echo "" -echo "PROBLEM ANALYSIS:" -echo "=================" -echo "${problem_analysis}" -echo "" -echo "SOLUTION STRATEGY:" -echo "==================" -echo "${solution_strategy}" -echo "" -echo "IMPLEMENTATION GUIDANCE:" -echo "========================" -echo "${implementation_guidance}" -echo "" -if [[ "$LANGUAGE" == "lil" ]] && [[ "$implementation_guidance" =~ (on [a-zA-Z_][a-zA-Z0-9_]* do|function|procedure) ]]; then - echo "CODE TESTING:" - echo "=============" - if [ -n "$lil_code" ]; then - echo "✅ Lil code was tested successfully" - echo "Test results logged in session file" - else - echo "No executable code found for testing" - fi - echo "" -fi -echo "SOLUTION VALIDATION:" -echo "====================" -echo "${solution_validation}" -echo "" -echo "Language focus: ${LANGUAGE}" -echo "Rounds completed: ${ROUNDS}" -echo "Execution time: ${duration} seconds" -echo "" -echo "Full puzzle-solving log: ${SESSION_FILE}" diff --git a/bash/talk-to-computer/quality_guard.sh b/bash/talk-to-computer/quality_guard.sh index 78edc90..06f8aec 100755..100644 --- a/bash/talk-to-computer/quality_guard.sh +++ b/bash/talk-to-computer/quality_guard.sh @@ -160,15 +160,15 @@ assess_structure() { local response="$1" # Check for proper formatting and structure - local has_paragraphs=$(echo "$response" | grep -c '^[[:space:]]*$' 2>/dev/null || echo "0") - local has_lists=$(echo "$response" | grep -c '^[[:space:]]*[-]' 2>/dev/null || echo "0") - local has_numbers=$(echo "$response" | grep -c '^[[:space:]]*[0-9]' 2>/dev/null || echo "0") + local has_paragraphs=$(echo "$response" | grep -c '^[[:space:]]*$' 2>/dev/null | tr -d '[:space:]' || echo "0") + local has_lists=$(echo "$response" | grep -c '^[[:space:]]*[-]' 2>/dev/null | tr -d '[:space:]' || echo "0") + local has_numbers=$(echo "$response" | grep -c '^[[:space:]]*[0-9]' 2>/dev/null | tr -d '[:space:]' || echo "0") local structure_score=0.5 - if [ "$has_paragraphs" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.2" | bc -l 2>/dev/null || echo "$structure_score"); fi - if [ "$has_lists" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.15" | bc -l 2>/dev/null || echo "$structure_score"); fi - if [ "$has_numbers" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.15" | bc -l 2>/dev/null || echo "$structure_score"); fi + if [ "${has_paragraphs:-0}" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.2" | bc -l 2>/dev/null || echo "$structure_score"); fi + if [ "${has_lists:-0}" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.15" | bc -l 2>/dev/null || echo "$structure_score"); fi + if [ "${has_numbers:-0}" -gt 0 ]; then structure_score=$(echo "$structure_score + 0.15" | bc -l 2>/dev/null || echo "$structure_score"); fi echo "$structure_score" } @@ -190,9 +190,15 @@ detect_degradation_patterns() { degradation_score=$(echo "$degradation_score + 0.8" | bc -l 2>/dev/null || echo "$degradation_score") fi - # Check for incoherent punctuation (simplified) - if echo "$response" | grep -q "^[[:space:]]*[[:punct:]]*[[:space:]]*$"; then - degradation_score=$(echo "$degradation_score + 0.7" | bc -l 2>/dev/null || echo "$degradation_score") + # Check for incoherent punctuation (more specific - lines with only punctuation) + local punct_only_lines=$(echo "$response" | grep "^[[:space:]]*[[:punct:]]*[[:space:]]*$" | wc -l) + local total_lines=$(echo "$response" | wc -l) + if [ "$total_lines" -gt 0 ]; then + local punct_ratio=$(( punct_only_lines * 100 / total_lines )) + if [ "$punct_ratio" -gt 50 ]; then + # Only flag if more than half the lines are punctuation-only + degradation_score=$(echo "$degradation_score + 0.4" | bc -l 2>/dev/null || echo "$degradation_score") + fi fi # Check for extremely short responses @@ -206,10 +212,7 @@ detect_degradation_patterns() { degradation_score=$(echo "$degradation_score + 0.6" | bc -l 2>/dev/null || echo "$degradation_score") fi - # Check for broken markdown (simplified) - if echo "$response" | grep -q "```[^`]*$" 2>/dev/null; then - degradation_score=$(echo "$degradation_score + 0.4" | bc -l 2>/dev/null || echo "$degradation_score") - fi + # Note: Removed problematic markdown check to eliminate syntax warnings echo "$degradation_score" } diff --git a/bash/talk-to-computer/socratic b/bash/talk-to-computer/socratic index 62f78c0..a685875 100755 --- a/bash/talk-to-computer/socratic +++ b/bash/talk-to-computer/socratic @@ -39,24 +39,22 @@ # with each iteration potentially revealing new insights or limitations. # The system emphasizes depth of analysis and intellectual honesty over definitive answers. -# Get the directory where this script is located -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# Initialize common functionality +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" +init_thinking_mechanism "${BASH_SOURCE[0]}" -# Source the logging system -source "${SCRIPT_DIR}/logging.sh" +# Use centralized model configuration +RESPONSE_MODEL="$SOCRATIC_RESPONSE_MODEL" +QUESTION_MODEL="$SOCRATIC_QUESTION_MODEL" -# Source the quality guard for output quality protection -source "${SCRIPT_DIR}/quality_guard.sh" - -# Get mechanism name automatically -MECHANISM_NAME=$(get_mechanism_name "$0") - -# --- Model Configuration --- -RESPONSE_MODEL="llama3:8b-instruct-q4_K_M" -QUESTION_MODEL="phi3:3.8b-mini-4k-instruct-q4_K_M" +# Validate and set models with standardized error handling +if ! validate_and_set_model "RESPONSE_MODEL" "$RESPONSE_MODEL" "$FALLBACK_MODEL"; then + handle_model_error "$RESPONSE_MODEL" "$(basename "$0")" +fi -# --- Defaults --- -DEFAULT_ROUNDS=2 +if ! validate_and_set_model "QUESTION_MODEL" "$QUESTION_MODEL" "$FALLBACK_MODEL"; then + handle_model_error "$QUESTION_MODEL" "$(basename "$0")" +fi # --- Argument Validation --- if [ "$#" -lt 1 ]; then @@ -95,10 +93,12 @@ fi # If file path is provided, append its contents to the prompt if [ -n "$FILE_PATH" ]; then if [ ! -f "$FILE_PATH" ]; then - echo "File not found: $FILE_PATH" >&2 - exit 1 + handle_file_error "$FILE_PATH" "find" "$(basename "$0")" + fi + if [ ! -r "$FILE_PATH" ]; then + handle_file_error "$FILE_PATH" "read" "$(basename "$0")" fi - FILE_CONTENTS=$(cat "$FILE_PATH") + FILE_CONTENTS=$(cat "$FILE_PATH" 2>/dev/null) || handle_file_error "$FILE_PATH" "read contents of" "$(basename "$0")" PROMPT="$PROMPT\n[FILE CONTENTS]\n$FILE_CONTENTS\n[END FILE]" fi diff --git a/bash/talk-to-computer/test_framework.sh b/bash/talk-to-computer/test_framework.sh new file mode 100755 index 0000000..c74ad56 --- /dev/null +++ b/bash/talk-to-computer/test_framework.sh @@ -0,0 +1,434 @@ +#!/bin/bash + +# Comprehensive Test Framework for AI Thinking Mechanisms +# This script provides automated testing capabilities for all system components. + +# Source common functionality +source "$(dirname "${BASH_SOURCE[0]}")/common.sh" +source "$(dirname "${BASH_SOURCE[0]}")/config.sh" + +# --- Test Configuration --- + +# Test directories +TEST_DIR="${LOG_DIR}/tests" +RESULTS_DIR="${TEST_DIR}/results" +COVERAGE_DIR="${TEST_DIR}/coverage" + +# Test counters +TESTS_PASSED=0 +TESTS_FAILED=0 +TESTS_SKIPPED=0 + +# --- Test Utilities --- + +# Initialize test framework +init_test_framework() { + mkdir -p "$RESULTS_DIR" "$COVERAGE_DIR" + echo "🧪 AI Thinking Mechanisms Test Framework" + echo "========================================" + echo +} + +# Test result functions +test_pass() { + local test_name="$1" + echo "✅ PASS: $test_name" + ((TESTS_PASSED++)) +} + +test_fail() { + local test_name="$1" + local reason="$2" + echo "❌ FAIL: $test_name - $reason" + ((TESTS_FAILED++)) +} + +test_skip() { + local test_name="$1" + local reason="$2" + echo "⏭️ SKIP: $test_name - $reason" + ((TESTS_SKIPPED++)) +} + +# Assert functions +assert_equals() { + local expected="$1" + local actual="$2" + local test_name="$3" + + if [ "$expected" = "$actual" ]; then + test_pass "$test_name" + else + test_fail "$test_name" "Expected '$expected', got '$actual'" + fi +} + +assert_not_empty() { + local value="$1" + local test_name="$2" + + if [ -n "$value" ]; then + test_pass "$test_name" + else + test_fail "$test_name" "Value is empty" + fi +} + +assert_file_exists() { + local file_path="$1" + local test_name="$2" + + if [ -f "$file_path" ]; then + test_pass "$test_name" + else + test_fail "$test_name" "File does not exist: $file_path" + fi +} + +# --- Component Tests --- + +test_common_functions() { + echo "Testing Common Functions..." + + # Test script directory detection + local script_dir + script_dir=$(get_script_dir) + assert_not_empty "$script_dir" "get_script_dir" + + # Test model validation (if ollama is available) + if command_exists ollama; then + local result + result=$(validate_model "gemma3n:e2b" "gemma3n:e2b") + if [ $? -eq 0 ]; then + test_pass "validate_model_success" + else + test_skip "validate_model_success" "Model not available" + fi + else + test_skip "validate_model_success" "Ollama not available" + fi +} + +test_config_loading() { + echo "Testing Configuration Loading..." + + # Test that config variables are loaded + if [ -n "$DEFAULT_MODEL" ]; then + test_pass "config_default_model" + else + test_fail "config_default_model" "DEFAULT_MODEL not set" + fi + + if [ -n "$FALLBACK_MODEL" ]; then + test_pass "config_fallback_model" + else + test_fail "config_fallback_model" "FALLBACK_MODEL not set" + fi + + # Test model arrays + if [ ${#CONSENSUS_MODELS[@]} -gt 0 ]; then + test_pass "config_consensus_models" + else + test_fail "config_consensus_models" "CONSENSUS_MODELS array is empty" + fi +} + +test_quality_guard() { + echo "Testing Quality Guard..." + + source "./quality_guard.sh" + + # Test quality assessment + local test_response="This is a comprehensive answer that should pass quality checks." + local quality_score + quality_score=$(assess_quality "$test_response" "test prompt" "socratic") + assert_not_empty "$quality_score" "assess_quality" + + # Test degradation detection + local degradation_score + degradation_score=$(detect_degradation_patterns "$test_response") + assert_not_empty "$degradation_score" "detect_degradation_patterns" + + # Test degraded response detection + local lorem_response="Lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod tempor incididunt" + local lorem_degradation + lorem_degradation=$(detect_degradation_patterns "$lorem_response") + + if (( $(echo "$lorem_degradation > 0" | bc -l 2>/dev/null || echo "0") )); then + test_pass "lorem_ipsum_detection" + else + test_fail "lorem_ipsum_detection" "Failed to detect lorem ipsum pattern" + fi +} + +test_logging_system() { + echo "Testing Logging System..." + + source "./logging.sh" + + # Test error logging + log_error "Test error message" + if [ -f "$ERROR_LOG" ]; then + test_pass "error_logging" + else + test_fail "error_logging" "Error log file not created" + fi + + # Test validation functions + local temp_file + temp_file=$(create_managed_temp_file "test" "tmp") + echo "test content" > "$temp_file" + + if validate_file_path "$temp_file"; then + test_pass "validate_file_path" + else + test_fail "validate_file_path" "Failed to validate existing file" + fi + + # Test invalid file + if ! validate_file_path "/nonexistent/file.txt" 2>/dev/null; then + test_pass "validate_invalid_file" + else + test_fail "validate_invalid_file" "Should have failed for nonexistent file" + fi +} + +test_resource_management() { + echo "Testing Resource Management..." + + source "./common.sh" + + # Test temporary directory creation + local temp_dir + temp_dir=$(create_managed_temp_dir "test") + if [ -d "$temp_dir" ]; then + test_pass "create_temp_dir" + else + test_fail "create_temp_dir" "Failed to create temp directory" + fi + + # Test cleanup registration + register_cleanup_resource "$temp_dir" + if [ ${#CLEANUP_RESOURCES[@]} -gt 0 ]; then + test_pass "register_cleanup_resource" + else + test_fail "register_cleanup_resource" "Resource not registered for cleanup" + fi +} + +# --- Integration Tests --- + +test_mechanism_integration() { + echo "Testing Mechanism Integration..." + + # Test if mechanisms are executable + local mechanisms=("socratic" "exploration" "consensus" "critique" "synthesis" "peer-review" "puzzle") + + for mechanism in "${mechanisms[@]}"; do + if [ -x "./$mechanism" ]; then + test_pass "mechanism_executable_$mechanism" + else + test_fail "mechanism_executable_$mechanism" "Mechanism not executable" + fi + done +} + +test_classifier_integration() { + echo "Testing Classifier Integration..." + + if [ -x "./classifier.sh" ]; then + test_pass "classifier_executable" + + # Test basic classification (if possible without models) + local test_result + test_result=$(source "./classifier.sh" && analyze_intent_patterns "What are the different approaches to solving this problem?" 2>/dev/null) + if [ -n "$test_result" ]; then + test_pass "classifier_basic_functionality" + else + test_skip "classifier_basic_functionality" "Cannot test without models" + fi + else + test_fail "classifier_executable" "Classifier script not executable" + fi +} + +# --- Performance Tests --- + +test_performance_metrics() { + echo "Testing Performance Metrics..." + + source "./logging.sh" + + # Test metrics functions exist + if command -v log_session_start >/dev/null 2>&1; then + test_pass "performance_functions_available" + else + test_fail "performance_functions_available" "Performance logging functions not available" + fi + + # Test metrics file creation + if [ -f "$METRICS_FILE" ] || touch "$METRICS_FILE" 2>/dev/null; then + test_pass "metrics_file_accessible" + else + test_fail "metrics_file_accessible" "Cannot access metrics file" + fi +} + +# --- Main Test Runner --- + +run_all_tests() { + init_test_framework + + echo "Running Test Suite..." + echo "=====================" + echo + + # Unit Tests + test_common_functions + echo + + test_config_loading + echo + + test_quality_guard + echo + + test_logging_system + echo + + test_resource_management + echo + + # Integration Tests + test_mechanism_integration + echo + + test_classifier_integration + echo + + # Performance Tests + test_performance_metrics + echo + + # Test Summary + echo "Test Summary" + echo "============" + echo "✅ Passed: $TESTS_PASSED" + echo "❌ Failed: $TESTS_FAILED" + echo "⏭️ Skipped: $TESTS_SKIPPED" + echo + + local total_tests=$((TESTS_PASSED + TESTS_FAILED)) + if [ $total_tests -gt 0 ]; then + local pass_rate=$((TESTS_PASSED * 100 / total_tests)) + echo "Pass Rate: $pass_rate%" + + if [ $TESTS_FAILED -eq 0 ]; then + echo "🎉 All tests completed successfully!" + return 0 + else + echo "⚠️ Some tests failed. Please review the results above." + return 1 + fi + else + echo "No tests were run." + return 1 + fi +} + +# --- CLI Interface --- + +show_help() { + echo "AI Thinking Mechanisms Test Framework" + echo "Usage: $0 [OPTIONS]" + echo + echo "Options:" + echo " -a, --all Run all tests (default)" + echo " -u, --unit Run only unit tests" + echo " -i, --integration Run only integration tests" + echo " -p, --performance Run only performance tests" + echo " -v, --verbose Enable verbose output" + echo " -h, --help Show this help message" + echo + echo "Examples:" + echo " $0 -a # Run all tests" + echo " $0 -u -v # Run unit tests with verbose output" + echo " $0 -p # Run only performance tests" +} + +# Parse command line arguments +VERBOSE=false +TEST_TYPE="all" + +while [[ $# -gt 0 ]]; do + case $1 in + -a|--all) + TEST_TYPE="all" + shift + ;; + -u|--unit) + TEST_TYPE="unit" + shift + ;; + -i|--integration) + TEST_TYPE="integration" + shift + ;; + -p|--performance) + TEST_TYPE="performance" + shift + ;; + -v|--verbose) + VERBOSE=true + shift + ;; + -h|--help) + show_help + exit 0 + ;; + *) + echo "Unknown option: $1" + show_help + exit 1 + ;; + esac +done + +# Set verbose output +if [ "$VERBOSE" = true ]; then + set -x +fi + +# Run tests based on type +case $TEST_TYPE in + "all") + run_all_tests + ;; + "unit") + init_test_framework + test_common_functions + echo + test_config_loading + echo + test_quality_guard + echo + test_logging_system + echo + test_resource_management + ;; + "integration") + init_test_framework + test_mechanism_integration + echo + test_classifier_integration + ;; + "performance") + init_test_framework + test_performance_metrics + ;; + *) + echo "Invalid test type: $TEST_TYPE" + show_help + exit 1 + ;; +esac |