Jarian Cottingham
1271f0b21b
- Remove agent/agent.md (dev-time agent context dumps), .DS_Store, committed venv configs (pyvenv.cfg), 0-byte runtime cache - Remove hardcoded /home/userpath from cron_scraper feed lookup - Replace ad-hoc test_implementation.py with pytest tests/test_scraper_cache.py - ruff clean (33 fixes: bare excepts, unused Config, whitespace) - Root pyproject.toml (activates shared Gitea CI), MIT LICENSE, README Tests
Scraper
A Python-based web scraping system designed to collect financial news and articles from various sources using RSS feeds and automated scraping techniques.
Overview
The Scraper component is responsible for collecting financial news and articles from multiple sources including major news outlets, financial publications, and market analysis services. It uses RSS feeds to gather content and stores the articles in a structured directory hierarchy for easy access by other components of the system.
Project Structure
scraper/
├── rss_feeds.json # Configuration file with RSS feed URLs
├── scraper.py # Main scraping logic
├── requirements.txt # Python dependencies
├── dockerfile # Docker configuration
├── dockerfile-selenium # Dockerfile for selenium-based scraping
├── articles/ # Directory where scraped articles are stored
│ ├── Reuters – Business News/
│ │ ├── article1.txt
│ │ └── ...
│ ├── Associated Press – Business/
│ │ ├── article1.txt
│ │ └── ...
│ └── ...
├── processed_articles_cache.json # Cache of already processed articles
└── pyvenv.cfg # Python virtual environment configuration
Features
- RSS Feed Integration: Supports multiple financial news sources through RSS feeds
- Automated Scraping: Regularly fetches and processes new articles from configured feeds
- Structured Storage: Organizes articles in a directory structure by news outlet
- Duplicate Detection: Prevents re-processing of already collected articles
- Caching Mechanism: Maintains a cache of processed articles to optimize performance
RSS Feed Sources
The scraper supports 60+ news outlets including:
- Reuters – Business News
- Associated Press – Business
- Financial Times
- Forbes – Real-Time
- Wall Street Journal – U.S. Business
- Bloomberg – Surveillance Podcast
- CNN Money
- BBC News – Business
- And many more...
Usage
Running the Scraper
python scraper.py
Configuration
The scraper can be configured by modifying rss_feeds.json to:
- Add new news sources
- Update existing RSS feed URLs
- Remove sources that are no longer active
Article Storage
Articles are stored in articles/ directory with the following structure:
articles/
└── <News Outlet Name>/
├── article1.txt
├── article2.txt
└── ...
Where each article file contains the full text content of that news article.
Requirements
- Python 3.6+
- Selenium WebDriver (for certain scraping operations)
- Additional dependencies listed in
requirements.txt
License
This project is licensed under the MIT License.