89 lines
2.8 KiB
Markdown
89 lines
2.8 KiB
Markdown
# Scraper
|
||
|
||
A Python-based web scraping system designed to collect financial news and articles from various sources using RSS feeds and automated scraping techniques.
|
||
|
||
## Overview
|
||
|
||
The Scraper component is responsible for collecting financial news and articles from multiple sources including major news outlets, financial publications, and market analysis services. It uses RSS feeds to gather content and stores the articles in a structured directory hierarchy for easy access by other components of the system.
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
scraper/
|
||
├── rss_feeds.json # Configuration file with RSS feed URLs
|
||
├── scraper.py # Main scraping logic
|
||
├── requirements.txt # Python dependencies
|
||
├── dockerfile # Docker configuration
|
||
├── dockerfile-selenium # Dockerfile for selenium-based scraping
|
||
├── articles/ # Directory where scraped articles are stored
|
||
│ ├── Reuters – Business News/
|
||
│ │ ├── article1.txt
|
||
│ │ └── ...
|
||
│ ├── Associated Press – Business/
|
||
│ │ ├── article1.txt
|
||
│ │ └── ...
|
||
│ └── ...
|
||
├── processed_articles_cache.json # Cache of already processed articles
|
||
└── pyvenv.cfg # Python virtual environment configuration
|
||
```
|
||
|
||
## Features
|
||
|
||
- **RSS Feed Integration**: Supports multiple financial news sources through RSS feeds
|
||
- **Automated Scraping**: Regularly fetches and processes new articles from configured feeds
|
||
- **Structured Storage**: Organizes articles in a directory structure by news outlet
|
||
- **Duplicate Detection**: Prevents re-processing of already collected articles
|
||
- **Caching Mechanism**: Maintains a cache of processed articles to optimize performance
|
||
|
||
## RSS Feed Sources
|
||
|
||
The scraper supports 60+ news outlets including:
|
||
|
||
- Reuters – Business News
|
||
- Associated Press – Business
|
||
- Financial Times
|
||
- Forbes – Real-Time
|
||
- Wall Street Journal – U.S. Business
|
||
- Bloomberg – Surveillance Podcast
|
||
- CNN Money
|
||
- BBC News – Business
|
||
- And many more...
|
||
|
||
## Usage
|
||
|
||
### Running the Scraper
|
||
|
||
```bash
|
||
python scraper.py
|
||
```
|
||
|
||
### Configuration
|
||
|
||
The scraper can be configured by modifying `rss_feeds.json` to:
|
||
- Add new news sources
|
||
- Update existing RSS feed URLs
|
||
- Remove sources that are no longer active
|
||
|
||
### Article Storage
|
||
|
||
Articles are stored in `articles/` directory with the following structure:
|
||
|
||
```
|
||
articles/
|
||
└── <News Outlet Name>/
|
||
├── article1.txt
|
||
├── article2.txt
|
||
└── ...
|
||
```
|
||
|
||
Where each article file contains the full text content of that news article.
|
||
|
||
## Requirements
|
||
|
||
- Python 3.6+
|
||
- Selenium WebDriver (for certain scraping operations)
|
||
- Additional dependencies listed in `requirements.txt`
|
||
|
||
## License
|
||
|
||
This project is licensed under the MIT License. |