Embedding

An embedding service that converts text content into numerical vectors for machine learning and data analysis purposes. This component transforms financial news articles into vector representations that can be used for similarity comparisons, clustering, and other AI tasks.

Overview

The Embedding project provides a service for converting textual financial news content into dense vector representations (embeddings) using state-of-the-art natural language processing models. These embeddings capture semantic meaning and relationships between different pieces of financial content, enabling advanced analytics and machine learning applications.

Project Structure

embedding/
├── app.py                 # Main embedding application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
├── models/                # Pre-trained embedding models
│   ├── __init__.py
│   ├── sentence_transformer.py  # Sentence transformer implementation
│   └── model_loader.py          # Model loading utilities
├── processors/            # Text processing pipeline
│   ├── __init__.py
│   ├── text_cleaner.py          # Text cleaning and preprocessing
│   └── embedding_generator.py   # Embedding generation
└── data/                  # Processed embeddings storage
    ├── cache/
    └── outputs/

Features

Multiple Model Support: Integrates with various embedding models including BERT, Sentence-BERT, and other transformer-based models
Batch Processing: Efficient processing of large volumes of articles
Caching Mechanism: Caches generated embeddings to avoid reprocessing
Real-time Generation: Generates embeddings on-demand for new content
Vector Similarity: Computes similarity between different pieces of content
Storage Management: Organizes and stores embeddings efficiently

Endpoints

Embedding Generation

POST /api/embeddings/generate - Generate embeddings for text content
POST /api/embeddings/batch - Generate embeddings for multiple texts in batch
GET /api/embeddings/similarity - Calculate similarity between two pieces of text/content

Data Management

GET /api/embeddings/status - Check service status
GET /api/embeddings/models - List available embedding models
DELETE /api/embeddings/cache/clear - Clear the embedding cache

Configuration

Environment Variables

The embedding service supports configuration through environment variables:

EMBEDDING_MODEL - Name of the pre-trained model to use (default: "all-MiniLM-L6-v2")
CACHE_DIR - Directory for caching generated embeddings (default: "data/cache/")
MAX_WORKERS - Number of concurrent processing threads (default: 4)
LOG_LEVEL - Logging level (DEBUG, INFO, WARNING, ERROR)
EMBEDDING_DIMENSION - Dimension size for generated vectors (default: 384)

Installation

Install dependencies:

pip install -r requirements.txt

Set up environment variables (optional but recommended):

export EMBEDDING_MODEL="all-MiniLM-L6-v2"
export CACHE_DIR="./data/cache/"
export MAX_WORKERS=4

Run the embedding service:

python app.py

Usage Examples

Generate embeddings for a single text:

curl -X POST "http://localhost:5002/api/embeddings/generate" \
  -H "Content-Type: application/json" \
  -d '{"text":"The stock market showed strong performance today.","source":"Reuters"}'

Batch processing of multiple texts:

curl -X POST "http://localhost:5002/api/embeddings/batch" \
  -H "Content-Type: application/json" \
  -d '{"texts":["Article 1 content","Article 2 content"],"source":"Financial News"}'

Calculate similarity between two pieces of text:

curl -X GET "http://localhost:5002/api/embeddings/similarity?text1=stock%20market&text2=financial%20market"

Requirements

Python 3.6+
Transformer models (transformers, sentence-transformers)
Vector processing libraries (numpy, scikit-learn)
Additional dependencies listed in requirements.txt

License

This project is licensed under the MIT License.

4.2 KiB Raw Permalink Blame History