FactsDB

A service for extracting and storing facts from different types of materials. Users can onboard directories containing various file types (text, HTML, PDF, etc.) and extract structured facts using AI.

Features

• Multi-format Support: Extract facts from text files, HTML pages, PDF documents, and more
• AI Integration: Uses OpenAI compatible endpoint for fact extraction
• Database Storage: SQLite database with proper locking and persistence
• FTP Server: Secure FTP server with access control for file management
• Automated Processing: Scheduled fact extraction jobs running every 10 minutes
• REST API: Query facts and manage the system through REST endpoints
• Monitoring: Prometheus/Grafana compatible metrics collection
• Docker Deployment: Complete Docker support for easy deployment

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   CLI Interface │    │   FTP Server    │    │  Scheduler      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
                    ┌─────────────────────────────────┐
                    │        Main Application         │
                    │    ┌─────────────────────────┐    │
                    │    │     File Processor    │    │
                    │    │     AI Processor      │    │
                    │    │    Database Manager   │    │
                    │    │    Monitoring         │    │
                    └─────────────────────────────────┘
                                 │
                    ┌─────────────────────────────────┐
                    │        SQLite Database          │
                    └─────────────────────────────────┘

Installation

Prerequisites

• Python 3.8+
• Docker (for containerized deployment)

Setup

# Clone the repository
git clone http://git.example.com/jarianc/FactsDB.git
cd FactsDB

# Install dependencies
pip install -r requirements.txt

# Initialize database
python -m factsdb.main initdb

Usage

Command Line Interface

# Onboard a directory
python -m factsdb.cli onboard --directory /path/to/data --table-name news_facts

# View status
python -m factsdb.cli status

# View tables
python -m factsdb.cli tables

Start Services

# Start FTP server
python -m factsdb.main ftp

# Start API server
python -m factsdb.main api

# Start scheduler
python -m factsdb.main scheduler

REST API Endpoints

• GET /health - Health check
• GET /tables - Get all available tables
• GET /tables/<table_name> - Get all facts from a specific table
• POST /tables/<table_name>/query - Query facts with custom query
• GET /tables/<table_name>/count - Get record count for a table
• GET /fact/<id> - Get a specific fact by ID
• GET /search - Search across all tables
• GET /version - Get service version
• GET /metrics - Prometheus metrics
• GET /metrics/json - JSON metrics
• GET /stats - Detailed service statistics

Configuration

The service uses a configuration file (config.json) that can be customized:

{
    "database": {
        "path": "facts.db"
    },
    "ai_endpoint": {
        "url": "http://example.com:4000/v1/chat/completions",
        "auth_token": "111"
    },
    "ftp_server": {
        "host": "0.0.0.0",
        "port": 2121,
        "username": "factsdb",
        "password": "factsdb"
    },
    "scheduler": {
        "interval_minutes": 10
    }
}

Docker Deployment

# Build and run with Docker Compose
docker-compose up --build

# Or run individual services
docker build -t factsdb .
docker run -p 5000:5000 -p 2121:2121 factsdb

File Processing Pipeline

The service supports various file types:

• Text files (.txt, .md)
• HTML pages (.html, .htm)
• PDF documents (.pdf)
• Other formats (converted to text)

Monitoring

The service provides Prometheus-compatible metrics:

• Fact extraction count
• File processing count
• Error count
• Uptime monitoring

Security

• FTP server with username/password authentication
• Database locking for concurrent access
• Secure AI endpoint communication
• File access control in FTP server

License

MIT License