StockDocs — Scraper

RSS scraper for the StockDocs financial news platform: collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes, FT, WSJ...) and stores articles in a structured directory hierarchy for the processing pipeline.

Part of the StockDocs project family:

Repo	What it is
StockDocs	Processing core — article server, NLP analysis, embeddings
stockdocs-mcp	MCP server exposing the processed data to LLM clients

Project Structure

.
├── rss_feeds.json         # Configuration file with RSS feed URLs
├── scraper.py             # Main scraping logic
├── cron_scraper.py        # Scheduled scraping entry point
├── run_scraper.sh         # Run helper
├── requirements.txt       # Python dependencies
├── dockerfile             # Docker configuration
├── dockerfile-selenium    # Dockerfile for selenium-based scraping
├── articles/              # Directory where scraped articles are stored
│   ├── Reuters – Business News/
│   │   ├── article1.txt
│   │   └── ...
│   └── ...
└── tests/                 # Unit tests (article-processing cache)

Features

• RSS Feed Integration: Supports multiple financial news sources through RSS feeds
• Automated Scraping: Regularly fetches and processes new articles from configured feeds
• Structured Storage: Organizes articles in a directory structure by news outlet
• Duplicate Detection: Prevents re-processing of already collected articles
• Caching Mechanism: Maintains a cache of processed articles to optimize performance

RSS Feed Sources

The scraper supports 60+ news outlets including:

• Reuters – Business News
• Associated Press – Business
• Financial Times
• Forbes – Real-Time
• Wall Street Journal – U.S. Business
• Bloomberg – Surveillance Podcast
• CNN Money
• BBC News – Business
• And many more...

Usage

Running the Scraper

python scraper.py

Configuration

The scraper can be configured by modifying rss_feeds.json to:

• Add new news sources
• Update existing RSS feed URLs
• Remove sources that are no longer active

Article Storage

Articles are stored in articles/ directory with the following structure:

articles/
└── <News Outlet Name>/
    ├── article1.txt
    ├── article2.txt
    └── ...

Where each article file contains the full text content of that news article.

Requirements

• Python 3.6+
• Selenium WebDriver (for certain scraping operations)
• Additional dependencies listed in requirements.txt

License

This project is licensed under the MIT License.