# StockDocs Project Overview

This document provides a comprehensive overview of the StockDocs repository, which contains multiple interconnected projects for processing, scraping, and serving financial articles and embeddings.

## Repository Structure

The repository contains 5 main projects:

1. **ai_processor/** - AI processing component
2. **articleServer/** - Article serving component  
3. **embedding/** - Embedding functionality
4. **MCPServer/** - MCP server component
5. **scraper/** - Web scraping component

## Project Details

### 1. ai_processor/
The AI processing component handles artificial intelligence operations for processing articles and generating insights.

**Key Files:**
- `ai_processor.py` - Main AI processing logic
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container configuration

**Purpose:** Processes articles using AI models to extract key information, generate summaries, and create embeddings.

**Detailed Structure:**
```
ai_processor/
├── app.py                 # Main AI processing application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration  
├── README.md              # This file
├── models/                # Machine learning models and NLP components
│   ├── __init__.py
│   ├── sentiment_analyzer.py  # Sentiment analysis module
│   ├── topic_classifier.py    # News categorization module
│   └── entity_extractor.py    # Named entity recognition
├── processors/            # Article processing pipelines
│   ├── __init__.py
│   ├── text_processor.py      # Text cleaning and preprocessing
│   └── analysis_pipeline.py   # Full analysis pipeline
└── data/                  # Processed data storage  
    ├── insights/
    └── reports/
```

**Endpoints:**
- POST `/api/analyze/article` - Analyze a single article for insights
- POST `/api/analyze/batch` - Process multiple articles in batch mode
- GET `/api/analyze/status/{task_id}` - Check processing status
- GET `/api/insights/latest` - Get latest analysis insights  
- GET `/api/insights/articles/{article_path}` - Get insights for specific article
- GET `/api/reports/generate` - Generate comprehensive market analysis report
- GET `/api/models` - List available AI models
- POST `/api/models/update` - Update or retrain models with new data

### 2. articleServer/
The article serving component provides an API for accessing processed articles.

**Key Files:**
- `app.py` - Main Flask application
- `run_server.py` - Server startup script
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container configuration

**Purpose:** Exposes processed articles through a REST API for client applications to consume.

**Detailed Structure:**
```
articleServer/
├── app.py                 # Main Flask application
├── run_server.py          # Server startup script
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
└── templates/             # HTML templates (if any)
```

**Endpoints:**
- GET `/articles` - Get articles within time range
- GET `/article/content` - Get full article content by path
- GET `/outlets` - List all available news outlets
- GET `/health` - Health check endpoint

### 3. embedding/
The embedding functionality handles vector embeddings for articles and documents.

**Key Files:**
- `embedder.py` - Embedding generation logic
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container configuration

**Purpose:** Converts articles into vector embeddings for semantic search and similarity operations.

**Detailed Structure:**
```
embedding/
├── app.py                 # Main embedding application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
├── models/                # Pre-trained embedding models
│   ├── __init__.py
│   ├── sentence_transformer.py  # Sentence transformer implementation
│   └── model_loader.py          # Model loading utilities
├── processors/            # Text processing pipeline
│   ├── __init__.py
│   ├── text_cleaner.py          # Text cleaning and preprocessing
│   └── embedding_generator.py   # Embedding generation
└── data/                  # Processed embeddings storage
    ├── cache/
    └── outputs/
```

**Endpoints:**
- POST `/api/embeddings/generate` - Generate embeddings for text content
- POST `/api/embeddings/batch` - Generate embeddings for multiple texts in batch
- GET `/api/embeddings/similarity` - Calculate similarity between two pieces of text/content
- GET `/api/embeddings/status` - Check service status
- GET `/api/embeddings/models` - List available embedding models
- DELETE `/api/embeddings/cache/clear` - Clear the embedding cache

### 4. MCPServer/
The MCP (Model Context Protocol) server component provides external API access.

**Key Files:**
- `server.py` - Main MCP server implementation
- `openapi.json` - API specification
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container configuration

**Purpose:** Exposes functionality through the Model Context Protocol for integration with other systems.

**Detailed Structure:**
```
MCPServer/
├── server.py              # Main Flask application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration
├── README.md              # This file
└── api/                   # API endpoints and handlers
    ├── __init__.py
    ├── stock_analysis.py   # Stock analysis functions
    └── news_processing.py  # News processing functions
```

**Endpoints:**
- GET `/api/stock/metrics` - Get financial metrics for a stock
- GET `/api/stock/history` - Get historical price data
- POST `/api/stock/analyze` - Perform comprehensive stock analysis
- GET `/api/news` - Retrieve news articles related to stocks
- GET `/api/news/outlets` - List available news sources
- POST `/api/news/process` - Process and categorize news content
- GET `/api/data/refresh` - Refresh data from sources
- GET `/api/status` - Server health check

### 5. scraper/
The web scraping component collects articles from various sources.

**Key Files:**
- `scraper.py` - Main scraping logic
- `rss_feeds.json` - RSS feed configuration
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container configuration

**Purpose:** Collects financial articles from various sources including RSS feeds and web scraping.

**Detailed Structure:**
```
scraper/
├── rss_feeds.json         # Configuration file with RSS feed URLs
├── scraper.py             # Main scraping logic
├── requirements.txt       # Python dependencies  
├── dockerfile             # Docker configuration
├── dockerfile-selenium    # Dockerfile for selenium-based scraping
├── articles/              # Directory where scraped articles are stored
│   ├── Reuters – Business News/
│   │   ├── article1.txt
│   │   └── ...
│   ├── Associated Press – Business/
│   │   ├── article1.txt  
│   │   └── ...
│   └── ...
├── processed_articles_cache.json  # Cache of already processed articles
└── pyvenv.cfg             # Python virtual environment configuration
```

**Features:**
- RSS Feed Integration: Supports multiple financial news sources through RSS feeds
- Automated Scraping: Regularly fetches and processes new articles from configured feeds  
- Structured Storage: Organizes articles in a directory structure by news outlet
- Duplicate Detection: Prevents re-processing of already collected articles
- Caching Mechanism: Maintains a cache of processed articles to optimize performance

## Getting Started

### Prerequisites
- Docker installed
- Python 3.8+
- Git

### Setup Instructions

1. **Clone the repository:**
   ```bash
   git clone <repository-url>
   cd StockDocs
   ```

2. **Build and run containers:**
   ```bash
   docker-compose up --build
   ```

3. **Project-specific setup:**
   - Each project has its own `README.md` with detailed setup instructions
   - Check individual project directories for specific requirements

## Project Dependencies

### Common Dependencies
- Python 3.8+
- Docker
- Various Python packages (listed in requirements.txt files)

### Inter-project Relationships
- `scraper/` feeds articles to `ai_processor/`
- `ai_processor/` generates embeddings that `embedding/` processes
- `articleServer/` serves articles processed by `ai_processor/`
- `MCPServer/` provides API access to all components

## Development Workflow

1. **Start all services:**
   ```bash
   docker-compose up --build
   ```

2. **Work with individual projects:**
   - Navigate to project directory
   - Check `README.md` for specific instructions
   - Make changes and rebuild as needed

3. **Testing:**
   - Each project includes its own testing setup
   - Integration tests may be needed for cross-project functionality

## API Endpoints

### articleServer/
- `/articles` - Get all articles
- `/articles/<id>` - Get specific article
- `/search` - Search articles by query

### MCPServer/
- Exposes various endpoints through Model Context Protocol
- See `openapi.json` for complete specification

## Configuration

### Environment Variables
Each project may require specific environment variables. Check individual `README.md` files for details.

### Data Storage
- Articles are stored in the scraper component
- Processed data flows through the ai_processor
- Embeddings are generated and stored in the embedding component

## Troubleshooting

### Common Issues
1. **Docker build failures:** Ensure Docker is running and check `Dockerfile` syntax
2. **Python dependency issues:** Run `pip install -r requirements.txt` in each project
3. **Port conflicts:** Check `docker-compose.yml` for port mappings
4. **Service startup issues:** Check individual project logs

### Logs
- View logs with `docker-compose logs <service-name>`
- Check individual project logs for detailed error information

## Contributing

1. Fork the repository
2. Create feature branch
3. Make changes
4. Test thoroughly
5. Submit pull request

## Support

For issues or questions, please check:
- Individual project README.md files
- Docker logs for runtime errors
- GitHub issues for known problems