first commit

README.md

@@ -0,0 +1,133 @@
+# NewsArchiver
+
+A news article archiving system that monitors RSS feeds and stores both raw HTML snapshots and parsed article content.
+
+## Features
+
+- Monitors 37 news websites via RSS feeds (business + technology)
+- High-fidelity web page archiving using SingleFile CLI
+- Content extraction with Trafilatura
+- SQLite database for caching and duplicate detection
+- Flask web interface for browsing archived content
+- Background scheduler for automated daily archiving (via cron)
+
+## Quick Start
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Setup cron job (recommended for production)
+./setup_cron.sh
+
+# Run archiver (one-time)
+python run_archiver.py --run
+
+# Start web interface
+python run_archiver.py --serve
+```
+
+## Setup Scripts
+
+- `setup_cron.sh` - Configure cron job for automated archiving every 15 minutes
+- `stop_services.sh` - Stop all NewsArchiver services gracefully
+
+## Cron Job Setup
+
+For production use, run the setup script:
+
+```bash
+./setup_cron.sh
+```
+
+This will:
+- Install dependencies if needed
+- Start the web server
+- Configure cron to run archiver every 15 minutes
+- Log output to `/tmp/newsarchiver_cron.log`
+
+The cron job runs independently of your terminal session, ensuring reliability.
+
+## Project Structure
+
+```
+NewsArchiver/
+├── run_archiver.py          # Main CLI entry point
+├── scheduler.py             # Background scheduler (APScheduler)
+├── rss_processor.py         # RSS feed fetching & caching
+├── content_extractor.py     # HTML parsing & content extraction
+├── storage_manager.py       # SQLite storage & file organization
+├── archive_engine.py        # Orchestration engine
+├── web_interface.py         # Flask web server
+├── singlefile_archive.py    # SingleFile CLI integration
+├── rss_feeds.json           # RSS feed configuration (37 feeds)
+├── archival_data/           # Archive output directory
+│   ├── cache.db            # SQLite cache database
+│   ├── processing.log      # Processing logs
+│   └── websites/           # Archived news sources
+└── templates/               # Flask templates
+```
+
+## Usage
+
+### Archive All Sources
+
+```bash
+python run_archiver.py --run
+```
+
+### Background Scheduler
+
+**Recommended:** Use the setup script to configure cron:
+
+```bash
+./setup_cron.sh
+```
+
+**Manual approach** (not recommended - can be killed by terminal disconnect):
+
+```bash
+# Run every 15 minutes
+python run_archiver.py --interval 15
+
+# Run every hour
+python run_archiver.py --interval 60
+```
+
+### Web Interface
+
+```bash
+python run_archiver.py --serve
+
+# Visit http://localhost:5000
+```
+
+## Database
+
+The SQLite database (`archival_data/cache.db`) stores:
+
+- **articles**: 2378+ archived articles with metadata
+- **article_archives**: Mapping of article URLs to archived HTML files
+- **processing_log**: Processing history and errors
+
+## Supported Sources
+
+**Business/Finance (29):** Associated Press, BBC News, The Guardian, The Economist, Financial Times, Fortune, Barchart News, Investor's Business Daily, Financial Samurai, MoneyWeek, Finance Monthly, European Financial Review, World Finance, Fox Business, FinanceAsia, CNBC, Markets Insider, Economy Watch, CFI.co, MarketWatch, Wall Street Journal, Investing.com, International Business Times, Seeking Alpha, Motley Fool, TheStreet, MarketBeat, Money, Global Finance Magazine
+
+**Technology (8):** 404 Media, Mac Rumors, The Verge, TechCrunch, WIRED, Hacker News, ZDNet, Engadget
+
+**Total:** 37 sources (see `rss_feeds.json`)
+
+## Requirements
+
+- Python 3.8+
+- Flask, Trafilatura, feedparser, APScheduler, requests, beautifulsoup4
+- SingleFile CLI (optional, for web page archiving)
+
+## Stopping Services
+
+To stop all NewsArchiver services:
+
+```bash
+./stop_services.sh
+```