# StockDocs — Scraper

RSS scraper for the StockDocs financial news platform: collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes, FT, WSJ...) and stores articles in a structured directory hierarchy for the processing pipeline.

Part of the StockDocs project family:

| Repo | What it is |
|------|------------|
| [StockDocs](https://git.jarianc.com/jarianc/StockDocs) | Processing core — article server, NLP analysis, embeddings |
| [stockdocs-mcp](https://git.jarianc.com/jarianc/stockdocs-mcp) | MCP server exposing the processed data to LLM clients |

## Project Structure

```
.
├── rss_feeds.json         # Configuration file with RSS feed URLs
├── scraper.py             # Main scraping logic
├── cron_scraper.py        # Scheduled scraping entry point
├── run_scraper.sh         # Run helper
├── requirements.txt       # Python dependencies
├── dockerfile             # Docker configuration
├── dockerfile-selenium    # Dockerfile for selenium-based scraping
├── articles/              # Directory where scraped articles are stored
│   ├── Reuters – Business News/
│   │   ├── article1.txt
│   │   └── ...
│   └── ...
└── tests/                 # Unit tests (article-processing cache)
```

## Features

- **RSS Feed Integration**: Supports multiple financial news sources through RSS feeds
- **Automated Scraping**: Regularly fetches and processes new articles from configured feeds  
- **Structured Storage**: Organizes articles in a directory structure by news outlet
- **Duplicate Detection**: Prevents re-processing of already collected articles
- **Caching Mechanism**: Maintains a cache of processed articles to optimize performance

## RSS Feed Sources

The scraper supports 60+ news outlets including:

- Reuters – Business News
- Associated Press – Business  
- Financial Times
- Forbes – Real-Time
- Wall Street Journal – U.S. Business
- Bloomberg – Surveillance Podcast
- CNN Money
- BBC News – Business
- And many more...

## Usage

### Running the Scraper

```bash
python scraper.py
```

### Configuration

The scraper can be configured by modifying `rss_feeds.json` to:
- Add new news sources
- Update existing RSS feed URLs  
- Remove sources that are no longer active

### Article Storage

Articles are stored in `articles/` directory with the following structure:

```
articles/
└── <News Outlet Name>/
    ├── article1.txt
    ├── article2.txt
    └── ...
```

Where each article file contains the full text content of that news article.

## Requirements

- Python 3.6+
- Selenium WebDriver (for certain scraping operations)
- Additional dependencies listed in `requirements.txt`

## License

This project is licensed under the MIT License.