StockDocs/README.md

# StockDocs

A comprehensive financial news analysis platform that combines web scraping, AI processing, and data embedding to provide actionable insights from financial news sources.

## Overview

StockDocs is a sophisticated system designed to collect, process, and analyze financial news from multiple sources. The platform consists of several interconnected components that work together to transform raw news content into valuable market intelligence.

## Project Components

### 1. Scraper
- Web scraping system for collecting financial news via RSS feeds
- Organizes articles by news source in a structured directory hierarchy
- Supports 60+ financial news outlets including Reuters, Bloomberg, Forbes, and more

### 2. Article Server 
- Flask-based HTTP server providing access to collected articles
- Query articles by time range and news outlet filters
- Retrieve full article content by file path
- Exposes RESTful API for external applications

### 3. AI Processor
- Natural language processing engine for analyzing news content
- Performs sentiment analysis, topic classification, and entity extraction
- Generates actionable insights from financial articles
- Supports batch processing of large volumes of content

### 4. Embedding Service
- Converts text content into numerical vector representations
- Enables semantic similarity comparisons between articles
- Supports various transformer-based models for high-quality embeddings
- Provides caching mechanism to optimize performance

### 5. MCPServer
- Market Capitalization Processor Server
- Provides financial data processing and API interface
- Integrates with stock analysis and news processing functions
- Serves as backend service for external access to financial data

## Architecture

```
+--------------+    +--------------+    +-------------------+
|   Scraper    |    | Article      |    |     AI            |
|  (RSS Feeds) |--> | Server       |--> | Processor         |
|              |    |              |    |                   |
+--------------+    +--------------+    +-------------------+
                           |                        |
                           v                        v
                    +--------------+    +-------------------+
                    | Embedding    |    |  MCPServer        |
                    | Service      |    | (Financial Data)  |
                    |              |    |                   |
                    +--------------+    +-------------------+
```

## Features

- **Multi-source News Collection**: Aggregates content from major financial news outlets
- **Real-time Processing**: Automated scraping and analysis pipeline 
- **Advanced Analytics**: NLP-powered sentiment and topic analysis
- **Semantic Search**: Vector-based similarity comparisons
- **RESTful APIs**: Clean interfaces for integration with external systems
- **Containerized Deployment**: Docker support for easy deployment

## Getting Started

### Prerequisites
- Python 3.6+
- Docker (for containerized deployment)
- Internet connection for RSS feed access

### Installation

1. Clone the repository:
```bash
git clone <repository-url>
cd StockDocs
```

2. Set up each component:
```bash
# For each component, follow specific installation instructions
cd scraper && pip install -r requirements.txt
cd articleServer && pip install -r requirements.txt
cd ai_processor && pip install -r requirements.txt
cd embedding && pip install -r requirements.txt
cd MCPServer && pip install -r requirements.txt
```

3. Configure environment variables as needed for each component

4. Run individual services:
```bash
python scraper/scraper.py          # Start scraping
python articleServer/run_server.py # Start article server  
python ai_processor/app.py         # Start AI processor
python embedding/app.py            # Start embedding service
python MCPServer/app.py            # Start MCP server
```

## Usage

### Data Collection
The scraper component automatically collects news from configured RSS feeds and stores articles in structured directories.

### API Access
Use the article server's RESTful APIs to access collected content:
```bash
# Get recent articles
curl "http://localhost:5008/articles?time_range=hour"

# Get article content
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
```

### Analysis
The AI processor and embedding service provide advanced analysis capabilities through their respective APIs.

## Deployment

Each component can be run independently or containerized using the provided Dockerfiles:
```bash
# Build and run each component in Docker
docker build -t stockdocs-scraper ./scraper
docker run -p 5000:5000 stockdocs-scraper

docker build -t stockdocs-article-server ./articleServer  
docker run -p 5008:5008 stockdocs-article-server

# Continue for other components...
```

## Tests

```bash
pip install -r scraper/requirements_clean.txt pytest
pytest tests/ -v
```

Unit tests cover the scraper's article-processing cache: load/save
roundtrips, processing status tracking, and progress computation.

## Requirements

- Python 3.9+
- Flask 2.3.3
- Various NLP and ML libraries
- Docker (for containerized deployment)

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Contributing

Contributions are welcome! Please read our contribution guidelines before submitting pull requests.

## Support

For support, please open an issue on the GitHub repository.