Scraper

A Python-based web scraping system designed to collect financial news and articles from various sources using RSS feeds and automated scraping techniques.

Overview

The Scraper component is responsible for collecting financial news and articles from multiple sources including major news outlets, financial publications, and market analysis services. It uses RSS feeds to gather content and stores the articles in a structured directory hierarchy for easy access by other components of the system.

Project Structure

scraper/
├── rss_feeds.json         # Configuration file with RSS feed URLs
├── scraper.py             # Main scraping logic
├── requirements.txt       # Python dependencies  
├── dockerfile             # Docker configuration
├── dockerfile-selenium    # Dockerfile for selenium-based scraping
├── articles/              # Directory where scraped articles are stored
│   ├── Reuters – Business News/
│   │   ├── article1.txt
│   │   └── ...
│   ├── Associated Press – Business/
│   │   ├── article1.txt  
│   │   └── ...
│   └── ...
├── processed_articles_cache.json  # Cache of already processed articles
└── pyvenv.cfg             # Python virtual environment configuration

Features

RSS Feed Integration: Supports multiple financial news sources through RSS feeds
Automated Scraping: Regularly fetches and processes new articles from configured feeds
Structured Storage: Organizes articles in a directory structure by news outlet
Duplicate Detection: Prevents re-processing of already collected articles
Caching Mechanism: Maintains a cache of processed articles to optimize performance

RSS Feed Sources

The scraper supports 60+ news outlets including:

Reuters – Business News
Associated Press – Business
Financial Times
Forbes – Real-Time
Wall Street Journal – U.S. Business
Bloomberg – Surveillance Podcast
CNN Money
BBC News – Business
And many more...

Usage

Running the Scraper

python scraper.py

Configuration

The scraper can be configured by modifying rss_feeds.json to:

Add new news sources
Update existing RSS feed URLs
Remove sources that are no longer active

Article Storage

Articles are stored in articles/ directory with the following structure:

articles/
└── <News Outlet Name>/
    ├── article1.txt
    ├── article2.txt
    └── ...

Where each article file contains the full text content of that news article.

Requirements

Python 3.6+
Selenium WebDriver (for certain scraping operations)
Additional dependencies listed in requirements.txt

License

This project is licensed under the MIT License.

2.8 KiB Raw Blame History Unescape Escape