158 lines
5.0 KiB
Markdown
158 lines
5.0 KiB
Markdown
# FactsDB
|
|
|
|
A service for extracting and storing facts from different types of materials. Users can onboard directories containing various file types (text, HTML, PDF, etc.) and extract structured facts using AI.
|
|
|
|
## Features
|
|
|
|
- **Multi-format Support**: Extract facts from text files, HTML pages, PDF documents, and more
|
|
- **AI Integration**: Uses OpenAI compatible endpoint for fact extraction
|
|
- **Database Storage**: SQLite database with proper locking and persistence
|
|
- **FTP Server**: Secure FTP server with access control for file management
|
|
- **Automated Processing**: Scheduled fact extraction jobs running every 10 minutes
|
|
- **REST API**: Query facts and manage the system through REST endpoints
|
|
- **Monitoring**: Prometheus/Grafana compatible metrics collection
|
|
- **Docker Deployment**: Complete Docker support for easy deployment
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ CLI Interface │ │ FTP Server │ │ Scheduler │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
│ │ │
|
|
└───────────────────────┼───────────────────────┘
|
|
│
|
|
┌─────────────────────────────────┐
|
|
│ Main Application │
|
|
│ ┌─────────────────────────┐ │
|
|
│ │ File Processor │ │
|
|
│ │ AI Processor │ │
|
|
│ │ Database Manager │ │
|
|
│ │ Monitoring │ │
|
|
└─────────────────────────────────┘
|
|
│
|
|
┌─────────────────────────────────┐
|
|
│ SQLite Database │
|
|
└─────────────────────────────────┘
|
|
```
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
- Python 3.8+
|
|
- Docker (for containerized deployment)
|
|
|
|
### Setup
|
|
```bash
|
|
# Clone the repository
|
|
git clone http://git.example.com/jarianc/FactsDB.git
|
|
cd FactsDB
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Initialize database
|
|
python -m factsdb.main initdb
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Command Line Interface
|
|
```bash
|
|
# Onboard a directory
|
|
python -m factsdb.cli onboard --directory /path/to/data --table-name news_facts
|
|
|
|
# View status
|
|
python -m factsdb.cli status
|
|
|
|
# View tables
|
|
python -m factsdb.cli tables
|
|
```
|
|
|
|
### Start Services
|
|
```bash
|
|
# Start FTP server
|
|
python -m factsdb.main ftp
|
|
|
|
# Start API server
|
|
python -m factsdb.main api
|
|
|
|
# Start scheduler
|
|
python -m factsdb.main scheduler
|
|
```
|
|
|
|
### REST API Endpoints
|
|
- `GET /health` - Health check
|
|
- `GET /tables` - Get all available tables
|
|
- `GET /tables/<table_name>` - Get all facts from a specific table
|
|
- `POST /tables/<table_name>/query` - Query facts with custom query
|
|
- `GET /tables/<table_name>/count` - Get record count for a table
|
|
- `GET /fact/<id>` - Get a specific fact by ID
|
|
- `GET /search` - Search across all tables
|
|
- `GET /version` - Get service version
|
|
- `GET /metrics` - Prometheus metrics
|
|
- `GET /metrics/json` - JSON metrics
|
|
- `GET /stats` - Detailed service statistics
|
|
|
|
## Configuration
|
|
|
|
The service uses a configuration file (`config.json`) that can be customized:
|
|
|
|
```json
|
|
{
|
|
"database": {
|
|
"path": "facts.db"
|
|
},
|
|
"ai_endpoint": {
|
|
"url": "http://example.com:4000/v1/chat/completions",
|
|
"auth_token": "111"
|
|
},
|
|
"ftp_server": {
|
|
"host": "0.0.0.0",
|
|
"port": 2121,
|
|
"username": "factsdb",
|
|
"password": "factsdb"
|
|
},
|
|
"scheduler": {
|
|
"interval_minutes": 10
|
|
}
|
|
}
|
|
```
|
|
|
|
## Docker Deployment
|
|
|
|
```bash
|
|
# Build and run with Docker Compose
|
|
docker-compose up --build
|
|
|
|
# Or run individual services
|
|
docker build -t factsdb .
|
|
docker run -p 5000:5000 -p 2121:2121 factsdb
|
|
```
|
|
|
|
## File Processing Pipeline
|
|
|
|
The service supports various file types:
|
|
- **Text files** (.txt, .md)
|
|
- **HTML pages** (.html, .htm)
|
|
- **PDF documents** (.pdf)
|
|
- **Other formats** (converted to text)
|
|
|
|
## Monitoring
|
|
|
|
The service provides Prometheus-compatible metrics:
|
|
- Fact extraction count
|
|
- File processing count
|
|
- Error count
|
|
- Uptime monitoring
|
|
|
|
## Security
|
|
|
|
- FTP server with username/password authentication
|
|
- Database locking for concurrent access
|
|
- Secure AI endpoint communication
|
|
- File access control in FTP server
|
|
|
|
## License
|
|
|
|
MIT License |