Jarian Cottingham
04f7365ac4
Move the RSS scraper into its own repository and the MCP data server into its own repository. This repo keeps the article server, AI processor, and embedding service. Compose and pyproject trimmed accordingly.
3.2 KiB
3.2 KiB
StockDocs
Financial news analysis platform: serves a collected article corpus over HTTP, runs NLP sentiment/topic/entity analysis, and builds transformer embeddings for semantic search.
Part of the StockDocs project family:
| Repo | What it is |
|---|---|
| stockdocs-scraper | RSS scraper — collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes...) into the article corpus |
| stockdocs-mcp | MCP server exposing the processed data to LLM clients |
This repository contains the processing core:
Project Components
1. Article Server
- Flask-based HTTP server providing access to the collected article corpus
- Query articles by time range and news outlet filters
- Retrieve full article content by file path
- Exposes RESTful API for external applications
2. AI Processor
- Natural language processing engine for analyzing news content
- Performs sentiment analysis, topic classification, and entity extraction
- Generates structured facts from financial articles
- Supports batch processing of large volumes of content
- Prometheus metrics for pipeline observability
3. Embedding Service
- Converts text content into numerical vector representations (ChromaDB)
- Enables semantic similarity comparisons between articles
- Transformer-based models for high-quality embeddings
- Caching mechanism to avoid re-embedding processed articles
Architecture
+---------------------+
| Article corpus | <- populated by stockdocs-scraper
+---------------------+
|
+------------+------------+
v v
+---------------+ +----------------+
| Article Server| | AI Processor |
| (Flask) | | (NLP) |
+---------------+ +----------------+
|
v
+----------------+
| Embedding |
| Service |
+----------------+
|
v
+----------------+
| ChromaDB |
+----------------+
Getting Started
Prerequisites
- Python 3.9+
- Docker (for containerized deployment)
- An article corpus directory (produced by stockdocs-scraper)
Installation
cd articleServer && pip install -r requirements.txt
cd ../ai_processor && pip install -r requirements.txt
cd ../embedding && pip install -r requirements.txt
Running
python articleServer/run_server.py # article server on :5008
python ai_processor/main.py # NLP pipeline
Usage
Query the article server:
# Get recent articles
curl "http://localhost:5008/articles?time_range=hour"
# Get article content
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
Deployment
All three services containerize with the included Dockerfiles; docker-compose.yml wires them onto a shared network with a shared articles volume.
License
This project is licensed under the MIT License — see the LICENSE file for details.