StockDocs/agent.md

StockDocs Project Overview

This document provides a comprehensive overview of the StockDocs repository, which contains multiple interconnected projects for processing, scraping, and serving financial articles and embeddings.

Repository Structure

The repository contains 5 main projects:

1. ai_processor/ - AI processing component
2. articleServer/ - Article serving component
3. embedding/ - Embedding functionality
4. MCPServer/ - MCP server component
5. scraper/ - Web scraping component

Project Details

1. ai_processor/

The AI processing component handles artificial intelligence operations for processing articles and generating insights.

Key Files:

• ai_processor.py - Main AI processing logic
• requirements.txt - Python dependencies
• Dockerfile - Container configuration

Purpose: Processes articles using AI models to extract key information, generate summaries, and create embeddings.

Detailed Structure:

ai_processor/
├── app.py                 # Main AI processing application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration  
├── README.md              # This file
├── models/                # Machine learning models and NLP components
│   ├── __init__.py
│   ├── sentiment_analyzer.py  # Sentiment analysis module
│   ├── topic_classifier.py    # News categorization module
│   └── entity_extractor.py    # Named entity recognition
├── processors/            # Article processing pipelines
│   ├── __init__.py
│   ├── text_processor.py      # Text cleaning and preprocessing
│   └── analysis_pipeline.py   # Full analysis pipeline
└── data/                  # Processed data storage  
    ├── insights/
    └── reports/

Endpoints:

• POST /api/analyze/article - Analyze a single article for insights
• POST /api/analyze/batch - Process multiple articles in batch mode
• GET /api/analyze/status/{task_id} - Check processing status
• GET /api/insights/latest - Get latest analysis insights
• GET /api/insights/articles/{article_path} - Get insights for specific article
• GET /api/reports/generate - Generate comprehensive market analysis report
• GET /api/models - List available AI models
• POST /api/models/update - Update or retrain models with new data

2. articleServer/

The article serving component provides an API for accessing processed articles.

Key Files:

• app.py - Main Flask application
• run_server.py - Server startup script
• requirements.txt - Python dependencies
• Dockerfile - Container configuration

Purpose: Exposes processed articles through a REST API for client applications to consume.

Detailed Structure:

articleServer/
├── app.py                 # Main Flask application
├── run_server.py          # Server startup script
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
└── templates/             # HTML templates (if any)

Endpoints:

• GET /articles - Get articles within time range
• GET /article/content - Get full article content by path
• GET /outlets - List all available news outlets
• GET /health - Health check endpoint

3. embedding/

The embedding functionality handles vector embeddings for articles and documents.

Key Files:

• embedder.py - Embedding generation logic
• requirements.txt - Python dependencies
• Dockerfile - Container configuration

Purpose: Converts articles into vector embeddings for semantic search and similarity operations.

Detailed Structure:

embedding/
├── app.py                 # Main embedding application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
├── models/                # Pre-trained embedding models
│   ├── __init__.py
│   ├── sentence_transformer.py  # Sentence transformer implementation
│   └── model_loader.py          # Model loading utilities
├── processors/            # Text processing pipeline
│   ├── __init__.py
│   ├── text_cleaner.py          # Text cleaning and preprocessing
│   └── embedding_generator.py   # Embedding generation
└── data/                  # Processed embeddings storage
    ├── cache/
    └── outputs/

Endpoints:

• POST /api/embeddings/generate - Generate embeddings for text content
• POST /api/embeddings/batch - Generate embeddings for multiple texts in batch
• GET /api/embeddings/similarity - Calculate similarity between two pieces of text/content
• GET /api/embeddings/status - Check service status
• GET /api/embeddings/models - List available embedding models
• DELETE /api/embeddings/cache/clear - Clear the embedding cache

4. MCPServer/

The MCP (Model Context Protocol) server component provides external API access.

Key Files:

• server.py - Main MCP server implementation
• openapi.json - API specification
• requirements.txt - Python dependencies
• Dockerfile - Container configuration

Purpose: Exposes functionality through the Model Context Protocol for integration with other systems.

Detailed Structure:

MCPServer/
├── server.py              # Main Flask application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
└── api/                   # API endpoints and handlers
    ├── __init__.py
    ├── stock_analysis.py   # Stock analysis functions
    └── news_processing.py  # News processing functions

Endpoints:

• GET /api/stock/metrics - Get financial metrics for a stock
• GET /api/stock/history - Get historical price data
• POST /api/stock/analyze - Perform comprehensive stock analysis
• GET /api/news - Retrieve news articles related to stocks
• GET /api/news/outlets - List available news sources
• POST /api/news/process - Process and categorize news content
• GET /api/data/refresh - Refresh data from sources
• GET /api/status - Server health check

5. scraper/

The web scraping component collects articles from various sources.

Key Files:

• scraper.py - Main scraping logic
• rss_feeds.json - RSS feed configuration
• requirements.txt - Python dependencies
• Dockerfile - Container configuration

Purpose: Collects financial articles from various sources including RSS feeds and web scraping.

Detailed Structure:

scraper/
├── rss_feeds.json         # Configuration file with RSS feed URLs
├── scraper.py             # Main scraping logic
├── requirements.txt       # Python dependencies  
├── dockerfile             # Docker configuration
├── dockerfile-selenium    # Dockerfile for selenium-based scraping
├── articles/              # Directory where scraped articles are stored
│   ├── Reuters – Business News/
│   │   ├── article1.txt
│   │   └── ...
│   ├── Associated Press – Business/
│   │   ├── article1.txt  
│   │   └── ...
│   └── ...
├── processed_articles_cache.json  # Cache of already processed articles
└── pyvenv.cfg             # Python virtual environment configuration

Features:

• RSS Feed Integration: Supports multiple financial news sources through RSS feeds
• Automated Scraping: Regularly fetches and processes new articles from configured feeds
• Structured Storage: Organizes articles in a directory structure by news outlet
• Duplicate Detection: Prevents re-processing of already collected articles
• Caching Mechanism: Maintains a cache of processed articles to optimize performance

Getting Started

Prerequisites

• Docker installed
• Python 3.8+
• Git

Setup Instructions

1. Clone the repository:

   git clone <repository-url>
   cd StockDocs
   

2. Build and run containers:

   docker-compose up --build
   

3. Project-specific setup:

   • Each project has its own README.md with detailed setup instructions
   • Check individual project directories for specific requirements

Project Dependencies

Common Dependencies

• Python 3.8+
• Docker
• Various Python packages (listed in requirements.txt files)

Inter-project Relationships

• scraper/ feeds articles to ai_processor/
• ai_processor/ generates embeddings that embedding/ processes
• articleServer/ serves articles processed by ai_processor/
• MCPServer/ provides API access to all components

Development Workflow

1. Start all services:

   docker-compose up --build
   

2. Work with individual projects:

   • Navigate to project directory
   • Check README.md for specific instructions
   • Make changes and rebuild as needed

3. Testing:

   • Each project includes its own testing setup
   • Integration tests may be needed for cross-project functionality

API Endpoints

articleServer/

• /articles - Get all articles
• /articles/<id> - Get specific article
• /search - Search articles by query

MCPServer/

• Exposes various endpoints through Model Context Protocol
• See openapi.json for complete specification

Configuration

Environment Variables

Each project may require specific environment variables. Check individual README.md files for details.

Authentication

The system now supports API key authentication for connections to the centralized AI service at http://example.com:4000. To enable authentication:

1. Set the AI_SERVICE_API_KEY environment variable with your API key
2. All connections to the AI service will automatically include the Authorization: Bearer {api_key} header

Data Storage

• Articles are stored in the scraper component
• Processed data flows through the ai_processor
• Embeddings are generated and stored in the embedding component

Troubleshooting

Common Issues

1. Docker build failures: Ensure Docker is running and check Dockerfile syntax
2. Python dependency issues: Run pip install -r requirements.txt in each project
3. Port conflicts: Check docker-compose.yml for port mappings
4. Service startup issues: Check individual project logs

Logs

• View logs with docker-compose logs <service-name>
• Check individual project logs for detailed error information

Contributing

1. Fork the repository
2. Create feature branch
3. Make changes
4. Test thoroughly
5. Submit pull request

Support

For issues or questions, please check:

• Individual project README.md files
• Docker logs for runtime errors
• GitHub issues for known problems