# StockDocs

Financial news analysis platform: serves a collected article corpus over HTTP, runs NLP sentiment/topic/entity analysis, and builds transformer embeddings for semantic search.

Part of the StockDocs project family:

| Repo | What it is |
|------|------------|
| [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper) | RSS scraper — collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes...) into the article corpus |
| [stockdocs-mcp](https://git.jarianc.com/jarianc/stockdocs-mcp) | MCP server exposing the processed data to LLM clients |

This repository contains the processing core:

## Project Components

### 1. Article Server
- Flask-based HTTP server providing access to the collected article corpus
- Query articles by time range and news outlet filters
- Retrieve full article content by file path
- Exposes RESTful API for external applications

### 2. AI Processor
- Natural language processing engine for analyzing news content
- Performs sentiment analysis, topic classification, and entity extraction
- Generates structured facts from financial articles
- Supports batch processing of large volumes of content
- Prometheus metrics for pipeline observability

### 3. Embedding Service
- Converts text content into numerical vector representations (ChromaDB)
- Enables semantic similarity comparisons between articles
- Transformer-based models for high-quality embeddings
- Caching mechanism to avoid re-embedding processed articles

## Architecture

```
                +---------------------+
                |   Article corpus    |  <- populated by stockdocs-scraper
                +---------------------+
                       |
          +------------+------------+
          v                        v
  +---------------+        +----------------+
  | Article Server|        |   AI Processor |
  |   (Flask)     |        |   (NLP)        |
  +---------------+        +----------------+
                       |
                       v
              +----------------+
              |  Embedding     |
              |  Service       |
              +----------------+
                       |
                       v
              +----------------+
              |  ChromaDB      |
              +----------------+
```

## Getting Started

### Prerequisites
- Python 3.9+
- Docker (for containerized deployment)
- An article corpus directory (produced by [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper))

### Installation

```bash
cd articleServer && pip install -r requirements.txt
cd ../ai_processor && pip install -r requirements.txt
cd ../embedding && pip install -r requirements.txt
```

### Running

```bash
python articleServer/run_server.py   # article server on :5008
python ai_processor/main.py          # NLP pipeline
```

## Usage

Query the article server:

```bash
# Get recent articles
curl "http://localhost:5008/articles?time_range=hour"

# Get article content
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
```

## Deployment

All three services containerize with the included Dockerfiles; `docker-compose.yml` wires them onto a shared network with a shared articles volume.

## License

This project is licensed under the MIT License — see the LICENSE file for details.