StockDocs/embedding

Embedding

An embedding service that converts text content into numerical vectors for machine learning and data analysis purposes. This component transforms financial news articles into vector representations that can be used for similarity comparisons, clustering, and other AI tasks.

Overview

The Embedding project provides a service for converting textual financial news content into dense vector representations (embeddings) using state-of-the-art natural language processing models. These embeddings capture semantic meaning and relationships between different pieces of financial content, enabling advanced analytics and machine learning applications.

Project Structure

embedding/
├── app.py                 # Main embedding application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
├── models/                # Pre-trained embedding models
│   ├── __init__.py
│   ├── sentence_transformer.py  # Sentence transformer implementation
│   └── model_loader.py          # Model loading utilities
├── processors/            # Text processing pipeline
│   ├── __init__.py
│   ├── text_cleaner.py          # Text cleaning and preprocessing
│   └── embedding_generator.py   # Embedding generation
└── data/                  # Processed embeddings storage
    ├── cache/
    └── outputs/

Features

• Multiple Model Support: Integrates with various embedding models including BERT, Sentence-BERT, and other transformer-based models
• Batch Processing: Efficient processing of large volumes of articles
• Caching Mechanism: Caches generated embeddings to avoid reprocessing
• Real-time Generation: Generates embeddings on-demand for new content
• Vector Similarity: Computes similarity between different pieces of content
• Storage Management: Organizes and stores embeddings efficiently

Endpoints

Embedding Generation

• POST /api/embeddings/generate - Generate embeddings for text content
• POST /api/embeddings/batch - Generate embeddings for multiple texts in batch
• GET /api/embeddings/similarity - Calculate similarity between two pieces of text/content

Data Management

• GET /api/embeddings/status - Check service status
• GET /api/embeddings/models - List available embedding models
• DELETE /api/embeddings/cache/clear - Clear the embedding cache

Configuration

Environment Variables

The embedding service supports configuration through environment variables:

• EMBEDDING_MODEL - Name of the pre-trained model to use (default: "all-MiniLM-L6-v2")
• CACHE_DIR - Directory for caching generated embeddings (default: "data/cache/")
• MAX_WORKERS - Number of concurrent processing threads (default: 4)
• LOG_LEVEL - Logging level (DEBUG, INFO, WARNING, ERROR)
• EMBEDDING_DIMENSION - Dimension size for generated vectors (default: 384)

Installation

1. Install dependencies:

pip install -r requirements.txt

2. Set up environment variables (optional but recommended):

export EMBEDDING_MODEL="all-MiniLM-L6-v2"
export CACHE_DIR="./data/cache/"
export MAX_WORKERS=4

3. Run the embedding service:

python app.py

Usage Examples

Generate embeddings for a single text:

curl -X POST "http://localhost:5002/api/embeddings/generate" \
  -H "Content-Type: application/json" \
  -d '{"text":"The stock market showed strong performance today.","source":"Reuters"}'

Batch processing of multiple texts:

curl -X POST "http://localhost:5002/api/embeddings/batch" \
  -H "Content-Type: application/json" \
  -d '{"texts":["Article 1 content","Article 2 content"],"source":"Financial News"}'

Calculate similarity between two pieces of text:

curl -X GET "http://localhost:5002/api/embeddings/similarity?text1=stock%20market&text2=financial%20market"

Requirements

• Python 3.6+
• Transformer models (transformers, sentence-transformers)
• Vector processing libraries (numpy, scikit-learn)
• Additional dependencies listed in requirements.txt

License

This project is licensed under the MIT License.