Jarian Cottingham
04f7365ac4
Move the RSS scraper into its own repository and the MCP data server into its own repository. This repo keeps the article server, AI processor, and embedding service. Compose and pyproject trimmed accordingly.
102 lines
3.2 KiB
Markdown
102 lines
3.2 KiB
Markdown
# StockDocs
|
|
|
|
Financial news analysis platform: serves a collected article corpus over HTTP, runs NLP sentiment/topic/entity analysis, and builds transformer embeddings for semantic search.
|
|
|
|
Part of the StockDocs project family:
|
|
|
|
| Repo | What it is |
|
|
|------|------------|
|
|
| [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper) | RSS scraper — collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes...) into the article corpus |
|
|
| [stockdocs-mcp](https://git.jarianc.com/jarianc/stockdocs-mcp) | MCP server exposing the processed data to LLM clients |
|
|
|
|
This repository contains the processing core:
|
|
|
|
## Project Components
|
|
|
|
### 1. Article Server
|
|
- Flask-based HTTP server providing access to the collected article corpus
|
|
- Query articles by time range and news outlet filters
|
|
- Retrieve full article content by file path
|
|
- Exposes RESTful API for external applications
|
|
|
|
### 2. AI Processor
|
|
- Natural language processing engine for analyzing news content
|
|
- Performs sentiment analysis, topic classification, and entity extraction
|
|
- Generates structured facts from financial articles
|
|
- Supports batch processing of large volumes of content
|
|
- Prometheus metrics for pipeline observability
|
|
|
|
### 3. Embedding Service
|
|
- Converts text content into numerical vector representations (ChromaDB)
|
|
- Enables semantic similarity comparisons between articles
|
|
- Transformer-based models for high-quality embeddings
|
|
- Caching mechanism to avoid re-embedding processed articles
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+---------------------+
|
|
| Article corpus | <- populated by stockdocs-scraper
|
|
+---------------------+
|
|
|
|
|
+------------+------------+
|
|
v v
|
|
+---------------+ +----------------+
|
|
| Article Server| | AI Processor |
|
|
| (Flask) | | (NLP) |
|
|
+---------------+ +----------------+
|
|
|
|
|
v
|
|
+----------------+
|
|
| Embedding |
|
|
| Service |
|
|
+----------------+
|
|
|
|
|
v
|
|
+----------------+
|
|
| ChromaDB |
|
|
+----------------+
|
|
```
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
- Python 3.9+
|
|
- Docker (for containerized deployment)
|
|
- An article corpus directory (produced by [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper))
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
cd articleServer && pip install -r requirements.txt
|
|
cd ../ai_processor && pip install -r requirements.txt
|
|
cd ../embedding && pip install -r requirements.txt
|
|
```
|
|
|
|
### Running
|
|
|
|
```bash
|
|
python articleServer/run_server.py # article server on :5008
|
|
python ai_processor/main.py # NLP pipeline
|
|
```
|
|
|
|
## Usage
|
|
|
|
Query the article server:
|
|
|
|
```bash
|
|
# Get recent articles
|
|
curl "http://localhost:5008/articles?time_range=hour"
|
|
|
|
# Get article content
|
|
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
|
|
```
|
|
|
|
## Deployment
|
|
|
|
All three services containerize with the included Dockerfiles; `docker-compose.yml` wires them onto a shared network with a shared articles volume.
|
|
|
|
## License
|
|
|
|
This project is licensed under the MIT License — see the LICENSE file for details.
|