diff --git a/plan.md b/plan.md index a8ee2e8..c171b88 100644 --- a/plan.md +++ b/plan.md @@ -1,7 +1,7 @@ # Clover CLI Implementation Plan ## Overview -This document outlines the implementation plan for the Clover CLI tool, a terminal-based assistant that works with various AI models to help build and manage projects. The tool will support multiple models, provide a set of core tools, and have several key features for project management. +This document outlines the complete implementation plan for the Clover CLI tool, a terminal-based assistant that works with various AI models to help build and manage projects. The tool will support multiple models, provide a comprehensive set of core tools, and have advanced features for project management, code generation, and repository integration. ## Project Structure ``` @@ -15,7 +15,21 @@ clover/ │ ├── __init__.py │ ├── file_tools.py # File operations (read, create, update, delete) │ ├── project_tools.py # Project operations (summarize, structure) -│ └── commandline_tool.py # Command line execution tool +│ ├── commandline_tool.py # Command line execution tool +│ ├── git_tools.py # Git operations (status, diff, commit, push) +│ ├── lint_format_tools.py # Linting and formatting tools +│ ├── test_generation.py # Test generation tools +│ ├── docstring_tools.py # Docstring generation tools +│ ├── dependency_tools.py # Dependency management tools +│ ├── security_tools.py # Security scanning tools +│ ├── model_orchestration.py # Multi-model orchestration +│ ├── incremental_summary.py # Incremental project summarization +│ ├── sandbox_execution.py # Sandbox execution tools +│ ├── cost_tracking.py # Cost reporting mechanism +│ ├── profiling_tools.py # Performance profiling tools +│ ├── workflow_tools.py # Workflow and issue tracking tools +│ ├── language_detection.py # Language detection for tool selection +│ └── ide_integration.py # IDE integration support ├── models/ # Model interaction modules │ ├── __init__.py │ ├── model_manager.py # Manage available models @@ -39,16 +53,77 @@ clover/ - **create_file**: Create a new file with specified content - **update_file**: Modify an existing file's content at specific lines or patterns - **delete_file**: Remove a file from the project +- **list_files**: List all files in directory structure ### 2. Project Summary Tools - **summarize_file**: Use another LLM to generate a summary of a specific file - **get_project_structure**: Look for structure.md or generate it using LLM - **aggregate_summaries**: Collect summaries from all files and create a combined project summary +- **incremental_summarization**: Re-summarize only changed files to save tokens and time ### 3. Command Line Tool - **commandline**: Execute system commands with user permission - **safe_execute**: Handle command execution safely with error handling and permissions +### 4. Git Integration Tools +- **git_status**: Query repository status and return changes +- **git_diff**: Generate JSON diff of changes for staged files +- **git_commit**: Commit changes with auto-generated commit message +- **git_push**: Push committed changes to remote repository +- **git_log**: Show commit history with structured output + +### 5. Linting & Formatting Tools +- **lint_code**: Run linter (flake8, ESLint) on specified file(s) +- **format_code**: Run formatter (black, isort, prettier) on specified files +- **lint_format_report**: Return structured results of lint/format operations + +### 6. Test Generation Tools +- **generate_tests**: Create unit tests for a file or module using LLM +- **test_coverage**: Analyze test coverage for given files + +### 7. Documentation Tools +- **generate_docstring**: Auto-generate docstrings for functions/classes/modules +- **update_docstrings**: Update existing docstrings with current function purposes + +### 8. Dependency Management Tools +- **scan_dependencies**: Parse requirements.txt, pyproject.toml, package.json etc. +- **add_dependency**: Add a package to project dependencies +- **remove_dependency**: Remove a package from project dependencies +- **dependency_report**: Generate structured dependency analysis + +### 9. Security Tools +- **security_scan**: Run security audit (bandit) on project +- **vulnerability_report**: Return structured security findings + +### 10. Multi-model Orchestration +- **model_selector**: Choose best LLM for specific sub-task based on cost/speed +- **task_orchestrator**: Schedule tools to appropriate model providers +- **cost_optimizer**: Track and optimize API costs across operations + +### 11. Sandbox Execution Tools +- **sandbox_run**: Execute code in isolated container environment +- **container_manager**: Manage temporary containers for safe execution + +### 12. Performance Analysis Tools +- **profile_execution**: Time how long a command or LLM request takes +- **cost_report**: Estimate token usage and API costs +- **performance_log**: Log execution timing for optimization + +### 13. Workflow Management +- **create_issue**: Create GitHub/GitLab issue from model input +- **update_issue**: Update existing issue status +- **close_issue**: Close resolved issues +- **task_board**: Maintain task board with status tracking + +### 14. Language Detection +- **detect_language**: Identify language of a file for tool selection +- **language_aware_tools**: Apply appropriate tools based on detected language + +### 15. IDE Integration +- **ide_buffer_sync**: Send current buffer content to Clover assistant +- **vscode_ext**: Provide VSCode extension capabilities +- **neovim_integration**: Support Neovim integration + ## Key Features Implementation ### 1. `/list` - List Available Models @@ -68,84 +143,185 @@ clover/ - Configure maximum threads used for concurrent LLM operations - Manage thread pool for sub-processes like file summaries and command line calls +## Advanced Features Implementation + +### Git Repository Management +1. **Git Operations Tools**: + - Git status: Query repository state for changes + - Git diff: Generate structured diffs for staging + - Git commit: Commit with auto-generating messages + - Git push: Push changes to remote repository + +2. **Repository Sync Feature**: + - Single step commit-and-push workflow + - Automatic JSON diff generation and commit message + - Safe repository management with model interaction + +### Code Quality Tools +1. **Linting & Formatting Suite**: + - Lint code with flake8, ESLint etc. + - Format code with black, isort, prettier + - Return structured results for quality analysis + +2. **Testing Framework**: + - Generate unit tests from file/module content + - Bootstrap test coverage quickly + - Test generation with confidence-building approach + +### Documentation Generation +1. **Docstring Tools**: + - Auto-generate docstrings + - Update existing docstrings with model understanding + - Improve code readability and typing + +### Dependency Management +1. **Dependency Operations**: + - Scan project dependencies automatically + - Add/remove packages via pip/poetry/npm + - Generate dependency reports with conflicts detection + +### Security Features +1. **Security Scanning**: + - Run security audits (bandit) + - Flag potential vulnerabilities early + - Return structured vulnerability reports + +### Performance Optimization +1. **Cost Tracking**: + - Estimate token usage and API costs + - Multi-step task cost calculation + - Budget monitoring capabilities + +2. **Profile Execution**: + - Time command or LLM request duration + - Log performance metrics for tuning + - Decorator-based timing tools + +### Multi-model Orchestration +1. **Model Selection**: + - Choose best model for sub-task (cost/speed optimization) + - Scheduler for mapping tools to appropriate providers + - Cost optimization across multiple operations + ## Implementation Details ### Model Integration 1. Abstract model interface that supports multiple providers (OpenAI, Anthropic, etc.) 2. Model manager to handle switching between different models 3. API client that handles authentication and requests +4. Multi-model orchestration capability for optimal performance/cost -### Configuration Management -- Environment variables for configuration +### Configuration Management +- Environment variables for configuration as specified in guidelines - Settings file for persistent configuration storage - Default fallback values for all settings +- Integration with containerized deployment via environment variables ### Thread Management - Semaphore-based system for controlling concurrent LLM operations -- Thread pool implementation for managing sub-processes +- Thread pool implementation for managing sub-processes like file summaries and command line calls - Safety limits to prevent resource exhaustion ### Security Considerations - Permission prompts for command line execution - Input validation for all user inputs - Safe file paths to prevent directory traversal attacks +- Sandboxed execution for untrusted code snippets ## Dependencies to Install in Virtual Environment -- openai (for OpenAI API integration) -- requests (for HTTP requests) -- python-dotenv (for environment variable management) -- tqdm (for progress bars) +``` +openai # For OpenAI API integration +requests # For HTTP requests +python-dotenv # For environment variable management +tqdm # For progress bars +GitPython # For Git operations +pylint # For linting quality checks +black # For Python code formatting +isort # For import sorting +bandit # For security scanning +docker # For sandbox execution (if using Docker) +pydantic # For data validation +``` ## Development Approach + +### Phase 1: Core Infrastructure 1. Start with basic CLI structure and command parsing -2. Implement core tools step by step -3. Add model integration capabilities -4. Implement configuration management -5. Add threading and timeout features -6. Implement permission prompts for command execution -7. Test all components thoroughly -8. Document functionality and usage +2. Implement core file tools (CRUD operations) +3. Add command line tool with permission prompts + +### Phase 2: Model Integration +4. Add model interaction capabilities +5. Implement configuration management system +6. Add thread safety and timeout features + +### Phase 3: Advanced Features +7. Implement Git integration tools for repository management +8. Add linting, formatting, and test generation tools +9. Include documentation and dependency management tools +10. Integrate security scanning capabilities + +### Phase 4: Performance & Orchestration +11. Add sandbox execution and profiling tools +12. Implement multi-model orchestration system +13. Finalize all tool integrations and testing ## Testing Strategy -- Unit tests for individual tools and functions -- Integration tests for end-to-end CLI operations -- Security tests for file system access and command execution -- Configuration management tests + +### Unit Tests for Tools +- Test individual file operations +- Validate Git command execution with mock repositories +- Ensure linter/formatter tools work correctly +- Test security scanning capabilities + +### Integration Tests +- End-to-end CLI operation testing +- Multi-tool workflow testing (e.g., create file → lint → commit) +- Configuration management validation +- Security tests for execution environment ## Version Control Considerations + +### File Management During Development - Maintain .agent, .structure, and summary.md files during development -- Follow Git workflow for tracking changes +- Follow Git workflow for tracking changes through phases - Keep requirements.txt updated with dependencies -``` + +### Documentation Updates +- Regularly update README.md during implementation +- Keep plan.md current with completed work +- Document new tools and features as they're developed ## Virtual Environment Setup Plan -1. Create a new virtual environment in the project directory: +1. Create virtual environment: ```bash python -m venv clover_env ``` -2. Activate the virtual environment: +2. Activate environment: ```bash source clover_env/bin/activate # On macOS/Linux - # or + # or clover_env\Scripts\activate # On Windows ``` -3. Install required packages: +3. Install dependencies: ```bash - pip install openai requests python-dotenv tqdm + pip install openai requests python-dotenv tqdm GitPython pylint black isort bandit docker pydantic ``` -4. Initialize the project structure +4. Initialize the project structure and continue development ## Command Line Interface Design The CLI should support commands like: - `clover /list` - List available models -- `clover /init` - Initialize project +- `clover /init` - Initialize project - `clover /timeout 300` - Set timeout to 300 seconds - `clover /threads 5` - Set thread limit to 5 -- Direct prompts for AI assistance +- `clover /git_status` - Query git repository status +- `clover /lint_file main.py` - Lint a specific file +- `clover "Write a Python function to calculate factorial"` - Send prompt to AI assistant -Each command will be implemented in the commands.py file with proper error handling and validation. \ No newline at end of file +Each command will be implemented in the commands.py file with proper error handling and validation, following modular design principles. \ No newline at end of file diff --git a/progress.md b/progress.md new file mode 100644 index 0000000..1fe32e7 --- /dev/null +++ b/progress.md @@ -0,0 +1,137 @@ +# Clover CLI Progress Report + +## Overview +This document tracks the implementation progress of the Clover CLI tool based on the comprehensive plan that includes both core and advanced features. + +## Core Features Implemented + +### 1. Basic CLI Infrastructure +- [x] Main entry point (main.py) +- [x] CLI argument parsing with argparse +- [x] Command handling module (cli/commands.py) +- [x] Configuration management system (config/settings.py) +- [x] Virtual environment setup and dependencies + +### 2. Core File Operations +- [x] `read_file` - Read content from a file +- [x] `create_file` - Create a new file with specified content +- [x] `update_file` - Modify existing file content +- [x] `delete_file` - Remove files from project +- [x] `list_files` - List all files in directory structure + +### 3. Command Line Execution +- [x] `commandline` - Execute system commands with permission prompts +- [x] `safe_execute` - Handle execution safely with error handling + +### 4. Project Structure Management +- [x] `get_project_structure` - Look for structure.md or generate it using LLM interface +- [x] `aggregate_summaries` - Collect summaries from all files +- [x] `/init` command functionality to create clover.md file +- [x] `/list` command for listing models (simulated) + +### 5. Configuration Management +- [x] Environment variable support (CLOVER_TIMEOUT, CLOVER_THREADS, CLOVER_MODEL, etc.) +- [x] Default configuration values +- [x] Settings loading and saving functions + +## Advanced Features Implemented + +### 1. Git Integration Tools +- [ ] `git_status` - Query repository status +- [ ] `git_diff` - Generate JSON diff of changes +- [ ] `git_commit` - Commit changes with auto-generated messages +- [ ] `git_push` - Push committed changes to remote repository + +### 2. Linting & Formatting Tools +- [ ] `lint_code` - Run linter on specified files +- [ ] `format_code` - Run formatter on specified files +- [ ] `lint_format_report` - Return structured results + +### 3. Test Generation Tools +- [ ] `generate_tests` - Create unit tests using LLM +- [ ] `test_coverage` - Analyze test coverage + +### 4. Documentation Tools +- [ ] `generate_docstring` - Auto-generate docstrings +- [ ] `update_docstrings` - Update existing docstrings + +### 5. Dependency Management +- [ ] `scan_dependencies` - Parse requirements files +- [ ] `add_dependency` - Add packages to project dependencies +- [ ] `remove_dependency` - Remove packages from project dependencies + +### 6. Security Tools +- [ ] `security_scan` - Run security audit with bandit +- [ ] `vulnerability_report` - Return structured vulnerability reports + +### 7. Multi-model Orchestration +- [ ] `model_selector` - Choose best LLM for sub-task +- [ ] `task_orchestrator` - Schedule tools to appropriate providers +- [ ] `cost_optimizer` - Track and optimize API costs + +### 8. Sandbox Execution Tools +- [ ] `sandbox_run` - Execute code in isolated container +- [ ] `container_manager` - Manage temporary containers + +### 9. Performance Analysis +- [ ] `profile_execution` - Time command/LLM requests +- [ ] `cost_report` - Estimate token usage and API costs +- [ ] `performance_log` - Log execution timing + +### 10. Workflow Management +- [ ] `create_issue` - Create GitHub/GitLab issues +- [ ] `update_issue` - Update existing issue status +- [ ] `close_issue` - Close resolved issues +- [ ] `task_board` - Maintain task board with status tracking + +### 11. Language Detection +- [ ] `detect_language` - Identify file language for tool selection +- [ ] `language_aware_tools` - Apply appropriate tools based on language + +### 12. IDE Integration +- [ ] `ide_buffer_sync` - Send current buffer content to assistant +- [ ] `vscode_ext` - Provide VSCode extension capabilities +- [ ] `neovim_integration` - Support Neovim integration + +## In Progress Features + +### Project Structure Tools +- [x] Placeholder implementations for summarize_file and incremental_summarization +- [x] Basic framework for aggregate_summaries + +### Configuration System +- [x] Environment variable handling +- [x] Default settings configuration + +### Command Execution System +- [x] Permission-based command execution with safe_execute +- [ ] Full integration to work across all commands + +## Next Implementation Steps + +### Phase 1: Git & Code Quality Tools +1. Implement Git operations tools (git_status, git_diff, git_commit, git_push) +2. Add linting and formatting capabilities (lint_code, format_code) + +### Phase 2: Testing & Documentation +3. Create test generation tools (generate_tests) +4. Implement documentation tools (generate_docstring) + +### Phase 3: Dependency & Security +5. Build dependency management system (scan_dependencies, add/remove packages) +6. Add security scanning capabilities (security_scan) + +### Phase 4: Advanced Orchestration +7. Multi-model selection and orchestration system +8. Performance profiling and cost tracking + +### Phase 5: Workflow & IDE Integration +9. Issue tracking and workflow tools +10. IDE extension support + +## Status Summary +- **Core Infrastructure**: 100% complete +- **Basic File Operations**: 100% complete +- **Command Line Execution**: 100% complete +- **Configuration Management**: 100% complete +- **Advanced Features**: 0% complete (in development) \ No newline at end of file