Jarian Cottingham
6b8a87f9ef
- Add rotating User-Agent pool (5 browser profiles) to all scrapers - Fix Playwright: use browser context for UA instead of broken set_extra_http_headers - Fix Selenium: set general.useragent.override preference - Fix newspaper4k: pass browser_user_agent to bypass bot detection - Add random delays (0.5-2s pre-fetch, 1-4s post-load) to mimic human behavior - Switch Reuters RSS from Google News proxy to official Reuters agency feed - Add legitimacy headers (Accept, Accept-Language, Connection) to Playwright - Apply all fixes to both scraper.py and cron_scraper.py
Scraper
A Python-based web scraping system designed to collect financial news and articles from various sources using RSS feeds and automated scraping techniques.
Overview
The Scraper component is responsible for collecting financial news and articles from multiple sources including major news outlets, financial publications, and market analysis services. It uses RSS feeds to gather content and stores the articles in a structured directory hierarchy for easy access by other components of the system.
Project Structure
scraper/
├── rss_feeds.json # Configuration file with RSS feed URLs
├── scraper.py # Main scraping logic
├── requirements.txt # Python dependencies
├── dockerfile # Docker configuration
├── dockerfile-selenium # Dockerfile for selenium-based scraping
├── articles/ # Directory where scraped articles are stored
│ ├── Reuters – Business News/
│ │ ├── article1.txt
│ │ └── ...
│ ├── Associated Press – Business/
│ │ ├── article1.txt
│ │ └── ...
│ └── ...
├── processed_articles_cache.json # Cache of already processed articles
└── pyvenv.cfg # Python virtual environment configuration
Features
- RSS Feed Integration: Supports multiple financial news sources through RSS feeds
- Automated Scraping: Regularly fetches and processes new articles from configured feeds
- Structured Storage: Organizes articles in a directory structure by news outlet
- Duplicate Detection: Prevents re-processing of already collected articles
- Caching Mechanism: Maintains a cache of processed articles to optimize performance
RSS Feed Sources
The scraper supports 60+ news outlets including:
- Reuters – Business News
- Associated Press – Business
- Financial Times
- Forbes – Real-Time
- Wall Street Journal – U.S. Business
- Bloomberg – Surveillance Podcast
- CNN Money
- BBC News – Business
- And many more...
Usage
Running the Scraper
python scraper.py
Configuration
The scraper can be configured by modifying rss_feeds.json to:
- Add new news sources
- Update existing RSS feed URLs
- Remove sources that are no longer active
Article Storage
Articles are stored in articles/ directory with the following structure:
articles/
└── <News Outlet Name>/
├── article1.txt
├── article2.txt
└── ...
Where each article file contains the full text content of that news article.
Requirements
- Python 3.6+
- Selenium WebDriver (for certain scraping operations)
- Additional dependencies listed in
requirements.txt
License
This project is licensed under the MIT License.