Update README.md with comprehensive documentation

README.md

@@ -0,0 +1,158 @@
+# FactsDB
+
+A service for extracting and storing facts from different types of materials. Users can onboard directories containing various file types (text, HTML, PDF, etc.) and extract structured facts using AI.
+
+## Features
+
+- **Multi-format Support**: Extract facts from text files, HTML pages, PDF documents, and more
+- **AI Integration**: Uses OpenAI compatible endpoint for fact extraction
+- **Database Storage**: SQLite database with proper locking and persistence
+- **FTP Server**: Secure FTP server with access control for file management
+- **Automated Processing**: Scheduled fact extraction jobs running every 10 minutes
+- **REST API**: Query facts and manage the system through REST endpoints
+- **Monitoring**: Prometheus/Grafana compatible metrics collection
+- **Docker Deployment**: Complete Docker support for easy deployment
+
+## Architecture
+
+```
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   CLI Interface │    │   FTP Server    │    │  Scheduler      │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+         │                       │                       │
+         └───────────────────────┼───────────────────────┘
+                                 │
+                    ┌─────────────────────────────────┐
+                    │        Main Application         │
+                    │    ┌─────────────────────────┐    │
+                    │    │     File Processor    │    │
+                    │    │     AI Processor      │    │
+                    │    │    Database Manager   │    │
+                    │    │    Monitoring         │    │
+                    └─────────────────────────────────┘
+                                 │
+                    ┌─────────────────────────────────┐
+                    │        SQLite Database          │
+                    └─────────────────────────────────┘
+```
+
+## Installation
+
+### Prerequisites
+- Python 3.8+
+- Docker (for containerized deployment)
+
+### Setup
+```bash
+# Clone the repository
+git clone http://git.example.com/jarianc/FactsDB.git
+cd FactsDB
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Initialize database
+python -m factsdb.main initdb
+```
+
+## Usage
+
+### Command Line Interface
+```bash
+# Onboard a directory
+python -m factsdb.cli onboard --directory /path/to/data --table-name news_facts
+
+# View status
+python -m factsdb.cli status
+
+# View tables
+python -m factsdb.cli tables
+```
+
+### Start Services
+```bash
+# Start FTP server
+python -m factsdb.main ftp
+
+# Start API server
+python -m factsdb.main api
+
+# Start scheduler
+python -m factsdb.main scheduler
+```
+
+### REST API Endpoints
+- `GET /health` - Health check
+- `GET /tables` - Get all available tables
+- `GET /tables/<table_name>` - Get all facts from a specific table
+- `POST /tables/<table_name>/query` - Query facts with custom query
+- `GET /tables/<table_name>/count` - Get record count for a table
+- `GET /fact/<id>` - Get a specific fact by ID
+- `GET /search` - Search across all tables
+- `GET /version` - Get service version
+- `GET /metrics` - Prometheus metrics
+- `GET /metrics/json` - JSON metrics
+- `GET /stats` - Detailed service statistics
+
+## Configuration
+
+The service uses a configuration file (`config.json`) that can be customized:
+
+```json
+{
+    "database": {
+        "path": "facts.db"
+    },
+    "ai_endpoint": {
+        "url": "http://example.com:4000/v1/chat/completions",
+        "auth_token": "111"
+    },
+    "ftp_server": {
+        "host": "0.0.0.0",
+        "port": 2121,
+        "username": "factsdb",
+        "password": "factsdb"
+    },
+    "scheduler": {
+        "interval_minutes": 10
+    }
+}
+```
+
+## Docker Deployment
+
+```bash
+# Build and run with Docker Compose
+docker-compose up --build
+
+# Or run individual services
+docker build -t factsdb .
+docker run -p 5000:5000 -p 2121:2121 factsdb
+```
+
+## File Processing Pipeline
+
+The service supports various file types:
+- **Text files** (.txt, .md)
+- **HTML pages** (.html, .htm)
+- **PDF documents** (.pdf)
+- **Other formats** (converted to text)
+
+## Monitoring
+
+The service provides Prometheus-compatible metrics:
+- Fact extraction count
+- File processing count
+- Error count
+- Uptime monitoring
+
+## Security
+
+- FTP server with username/password authentication
+- Database locking for concurrent access
+- Secure AI endpoint communication
+- File access control in FTP server
+
+## License
+
+MIT License