# FactsDB

A service for extracting and storing facts from different types of materials. Users can onboard directories containing various file types (text, HTML, PDF, etc.) and extract structured facts using AI.

## Features

- **Multi-format Support**: Extract facts from text files, HTML pages, PDF documents, and more
- **AI Integration**: Uses OpenAI compatible endpoint for fact extraction
- **Database Storage**: SQLite database with proper locking and persistence
- **FTP Server**: Secure FTP server with access control for file management
- **Automated Processing**: Scheduled fact extraction jobs running every 10 minutes
- **REST API**: Query facts and manage the system through REST endpoints
- **Monitoring**: Prometheus/Grafana compatible metrics collection
- **Docker Deployment**: Complete Docker support for easy deployment

## Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   CLI Interface │    │   FTP Server    │    │  Scheduler      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                    ┌─────────────────────────────────┐
                    │        Main Application         │
                    │    ┌─────────────────────────┐    │
                    │    │     File Processor    │    │
                    │    │     AI Processor      │    │
                    │    │    Database Manager   │    │
                    │    │    Monitoring         │    │
                    └─────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────┐
                    │        SQLite Database          │
                    └─────────────────────────────────┘
```

## Installation

### Prerequisites
- Python 3.8+
- Docker (for containerized deployment)

### Setup
```bash
# Clone the repository
git clone http://git.example.com/jarianc/FactsDB.git
cd FactsDB

# Install dependencies
pip install -r requirements.txt

# Initialize database
python -m factsdb.main initdb
```

## Usage

### Command Line Interface
```bash
# Onboard a directory
python -m factsdb.cli onboard --directory /path/to/data --table-name news_facts

# View status
python -m factsdb.cli status

# View tables
python -m factsdb.cli tables
```

### Start Services
```bash
# Start FTP server
python -m factsdb.main ftp

# Start API server
python -m factsdb.main api

# Start scheduler
python -m factsdb.main scheduler
```

### REST API Endpoints
- `GET /health` - Health check
- `GET /tables` - Get all available tables
- `GET /tables/<table_name>` - Get all facts from a specific table
- `POST /tables/<table_name>/query` - Query facts with custom query
- `GET /tables/<table_name>/count` - Get record count for a table
- `GET /fact/<id>` - Get a specific fact by ID
- `GET /search` - Search across all tables
- `GET /version` - Get service version
- `GET /metrics` - Prometheus metrics
- `GET /metrics/json` - JSON metrics
- `GET /stats` - Detailed service statistics

## Configuration

The service uses a configuration file (`config.json`) that can be customized:

```json
{
    "database": {
        "path": "facts.db"
    },
    "ai_endpoint": {
        "url": "http://example.com:4000/v1/chat/completions",
        "auth_token": "111"
    },
    "ftp_server": {
        "host": "0.0.0.0",
        "port": 2121,
        "username": "factsdb",
        "password": "factsdb"
    },
    "scheduler": {
        "interval_minutes": 10
    }
}
```

## Docker Deployment

```bash
# Build and run with Docker Compose
docker-compose up --build

# Or run individual services
docker build -t factsdb .
docker run -p 5000:5000 -p 2121:2121 factsdb
```

## File Processing Pipeline

The service supports various file types:
- **Text files** (.txt, .md)
- **HTML pages** (.html, .htm)
- **PDF documents** (.pdf)
- **Other formats** (converted to text)

## Monitoring

The service provides Prometheus-compatible metrics:
- Fact extraction count
- File processing count
- Error count
- Uptime monitoring

## Security

- FTP server with username/password authentication
- Database locking for concurrent access
- Secure AI endpoint communication
- File access control in FTP server

## License

MIT License