From 639f03fdb95252423b987e6d1e6155a1b4839370 Mon Sep 17 00:00:00 2001
From: Jarian Cottingham <jarianc@proton.me>
Date: Tue, 31 Mar 2026 09:37:06 -0500
Subject: [PATCH] first commit

---
 README.md | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..9d89e8e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,133 @@
+# NewsArchiver
+
+A news article archiving system that monitors RSS feeds and stores both raw HTML snapshots and parsed article content.
+
+## Features
+
+- Monitors 37 news websites via RSS feeds (business + technology)
+- High-fidelity web page archiving using SingleFile CLI
+- Content extraction with Trafilatura
+- SQLite database for caching and duplicate detection
+- Flask web interface for browsing archived content
+- Background scheduler for automated daily archiving (via cron)
+
+## Quick Start
+
+```bash
+# Install dependencies
+pip install -r requirements.txt
+
+# Setup cron job (recommended for production)
+./setup_cron.sh
+
+# Run archiver (one-time)
+python run_archiver.py --run
+
+# Start web interface
+python run_archiver.py --serve
+```
+
+## Setup Scripts
+
+- `setup_cron.sh` - Configure cron job for automated archiving every 15 minutes
+- `stop_services.sh` - Stop all NewsArchiver services gracefully
+
+## Cron Job Setup
+
+For production use, run the setup script:
+
+```bash
+./setup_cron.sh
+```
+
+This will:
+- Install dependencies if needed
+- Start the web server
+- Configure cron to run archiver every 15 minutes
+- Log output to `/tmp/newsarchiver_cron.log`
+
+The cron job runs independently of your terminal session, ensuring reliability.
+
+## Project Structure
+
+```
+NewsArchiver/
+├── run_archiver.py          # Main CLI entry point
+├── scheduler.py             # Background scheduler (APScheduler)
+├── rss_processor.py         # RSS feed fetching & caching
+├── content_extractor.py     # HTML parsing & content extraction
+├── storage_manager.py       # SQLite storage & file organization
+├── archive_engine.py        # Orchestration engine
+├── web_interface.py         # Flask web server
+├── singlefile_archive.py    # SingleFile CLI integration
+├── rss_feeds.json           # RSS feed configuration (37 feeds)
+├── archival_data/           # Archive output directory
+│   ├── cache.db            # SQLite cache database
+│   ├── processing.log      # Processing logs
+│   └── websites/           # Archived news sources
+└── templates/               # Flask templates
+```
+
+## Usage
+
+### Archive All Sources
+
+```bash
+python run_archiver.py --run
+```
+
+### Background Scheduler
+
+**Recommended:** Use the setup script to configure cron:
+
+```bash
+./setup_cron.sh
+```
+
+**Manual approach** (not recommended - can be killed by terminal disconnect):
+
+```bash
+# Run every 15 minutes
+python run_archiver.py --interval 15
+
+# Run every hour
+python run_archiver.py --interval 60
+```
+
+### Web Interface
+
+```bash
+python run_archiver.py --serve
+
+# Visit http://localhost:5000
+```
+
+## Database
+
+The SQLite database (`archival_data/cache.db`) stores:
+
+- **articles**: 2378+ archived articles with metadata
+- **article_archives**: Mapping of article URLs to archived HTML files
+- **processing_log**: Processing history and errors
+
+## Supported Sources
+
+**Business/Finance (29):** Associated Press, BBC News, The Guardian, The Economist, Financial Times, Fortune, Barchart News, Investor's Business Daily, Financial Samurai, MoneyWeek, Finance Monthly, European Financial Review, World Finance, Fox Business, FinanceAsia, CNBC, Markets Insider, Economy Watch, CFI.co, MarketWatch, Wall Street Journal, Investing.com, International Business Times, Seeking Alpha, Motley Fool, TheStreet, MarketBeat, Money, Global Finance Magazine
+
+**Technology (8):** 404 Media, Mac Rumors, The Verge, TechCrunch, WIRED, Hacker News, ZDNet, Engadget
+
+**Total:** 37 sources (see `rss_feeds.json`)
+
+## Requirements
+
+- Python 3.8+
+- Flask, Trafilatura, feedparser, APScheduler, requests, beautifulsoup4
+- SingleFile CLI (optional, for web page archiving)
+
+## Stopping Services
+
+To stop all NewsArchiver services:
+
+```bash
+./stop_services.sh
+```
\ No newline at end of file