Jarian Cottingham
724be90e7c
Carved out from the StockDocs monorepo. Collects financial news from 60+ outlets via RSS into a structured article corpus.
90 lines
2.7 KiB
Markdown
90 lines
2.7 KiB
Markdown
# StockDocs — Scraper
|
||
|
||
RSS scraper for the StockDocs financial news platform: collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes, FT, WSJ...) and stores articles in a structured directory hierarchy for the processing pipeline.
|
||
|
||
Part of the StockDocs project family:
|
||
|
||
| Repo | What it is |
|
||
|------|------------|
|
||
| [StockDocs](https://git.jarianc.com/jarianc/StockDocs) | Processing core — article server, NLP analysis, embeddings |
|
||
| [stockdocs-mcp](https://git.jarianc.com/jarianc/stockdocs-mcp) | MCP server exposing the processed data to LLM clients |
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
.
|
||
├── rss_feeds.json # Configuration file with RSS feed URLs
|
||
├── scraper.py # Main scraping logic
|
||
├── cron_scraper.py # Scheduled scraping entry point
|
||
├── run_scraper.sh # Run helper
|
||
├── requirements.txt # Python dependencies
|
||
├── dockerfile # Docker configuration
|
||
├── dockerfile-selenium # Dockerfile for selenium-based scraping
|
||
├── articles/ # Directory where scraped articles are stored
|
||
│ ├── Reuters – Business News/
|
||
│ │ ├── article1.txt
|
||
│ │ └── ...
|
||
│ └── ...
|
||
└── tests/ # Unit tests (article-processing cache)
|
||
```
|
||
|
||
## Features
|
||
|
||
- **RSS Feed Integration**: Supports multiple financial news sources through RSS feeds
|
||
- **Automated Scraping**: Regularly fetches and processes new articles from configured feeds
|
||
- **Structured Storage**: Organizes articles in a directory structure by news outlet
|
||
- **Duplicate Detection**: Prevents re-processing of already collected articles
|
||
- **Caching Mechanism**: Maintains a cache of processed articles to optimize performance
|
||
|
||
## RSS Feed Sources
|
||
|
||
The scraper supports 60+ news outlets including:
|
||
|
||
- Reuters – Business News
|
||
- Associated Press – Business
|
||
- Financial Times
|
||
- Forbes – Real-Time
|
||
- Wall Street Journal – U.S. Business
|
||
- Bloomberg – Surveillance Podcast
|
||
- CNN Money
|
||
- BBC News – Business
|
||
- And many more...
|
||
|
||
## Usage
|
||
|
||
### Running the Scraper
|
||
|
||
```bash
|
||
python scraper.py
|
||
```
|
||
|
||
### Configuration
|
||
|
||
The scraper can be configured by modifying `rss_feeds.json` to:
|
||
- Add new news sources
|
||
- Update existing RSS feed URLs
|
||
- Remove sources that are no longer active
|
||
|
||
### Article Storage
|
||
|
||
Articles are stored in `articles/` directory with the following structure:
|
||
|
||
```
|
||
articles/
|
||
└── <News Outlet Name>/
|
||
├── article1.txt
|
||
├── article2.txt
|
||
└── ...
|
||
```
|
||
|
||
Where each article file contains the full text content of that news article.
|
||
|
||
## Requirements
|
||
|
||
- Python 3.6+
|
||
- Selenium WebDriver (for certain scraping operations)
|
||
- Additional dependencies listed in `requirements.txt`
|
||
|
||
## License
|
||
|
||
This project is licensed under the MIT License. |