NewsArchiver

A news article archiving system that monitors RSS feeds and stores both raw HTML snapshots and parsed article content.

Features

• Monitors 37 news websites via RSS feeds (business + technology)
• High-fidelity web page archiving using SingleFile CLI
• Content extraction with Trafilatura
• SQLite database for caching and duplicate detection
• Flask web interface for browsing archived content
• Background scheduler for automated daily archiving (via cron)

Quick Start

# Install dependencies
pip install -r requirements.txt

# Setup cron job (recommended for production)
./setup_cron.sh

# Run archiver (one-time)
python run_archiver.py --run

# Start web interface
python run_archiver.py --serve

Setup Scripts

• setup_cron.sh - Configure cron job for automated archiving every 15 minutes
• stop_services.sh - Stop all NewsArchiver services gracefully

Cron Job Setup

For production use, run the setup script:

./setup_cron.sh

This will:

• Install dependencies if needed
• Start the web server
• Configure cron to run archiver every 15 minutes
• Log output to /tmp/newsarchiver_cron.log

The cron job runs independently of your terminal session, ensuring reliability.

Project Structure

NewsArchiver/
├── run_archiver.py          # Main CLI entry point
├── scheduler.py             # Background scheduler (APScheduler)
├── rss_processor.py         # RSS feed fetching & caching
├── content_extractor.py     # HTML parsing & content extraction
├── storage_manager.py       # SQLite storage & file organization
├── archive_engine.py        # Orchestration engine
├── web_interface.py         # Flask web server
├── singlefile_archive.py    # SingleFile CLI integration
├── rss_feeds.json           # RSS feed configuration (37 feeds)
├── archival_data/           # Archive output directory
│   ├── cache.db            # SQLite cache database
│   ├── processing.log      # Processing logs
│   └── websites/           # Archived news sources
└── templates/               # Flask templates

Usage

Archive All Sources

python run_archiver.py --run

Background Scheduler

Recommended: Use the setup script to configure cron:

./setup_cron.sh

Manual approach (not recommended - can be killed by terminal disconnect):

# Run every 15 minutes
python run_archiver.py --interval 15

# Run every hour
python run_archiver.py --interval 60

Web Interface

python run_archiver.py --serve

# Visit http://localhost:5000

Database

The SQLite database (archival_data/cache.db) stores:

• articles: 2378+ archived articles with metadata
• article_archives: Mapping of article URLs to archived HTML files
• processing_log: Processing history and errors

Supported Sources

Business/Finance (29): Associated Press, BBC News, The Guardian, The Economist, Financial Times, Fortune, Barchart News, Investor's Business Daily, Financial Samurai, MoneyWeek, Finance Monthly, European Financial Review, World Finance, Fox Business, FinanceAsia, CNBC, Markets Insider, Economy Watch, CFI.co, MarketWatch, Wall Street Journal, Investing.com, International Business Times, Seeking Alpha, Motley Fool, TheStreet, MarketBeat, Money, Global Finance Magazine

Technology (8): 404 Media, Mac Rumors, The Verge, TechCrunch, WIRED, Hacker News, ZDNet, Engadget

Total: 37 sources (see rss_feeds.json)

Requirements

• Python 3.8+
• Flask, Trafilatura, feedparser, APScheduler, requests, beautifulsoup4
• SingleFile CLI (optional, for web page archiving)

Docker Deployment

The NewsArchiver can be deployed using Docker for easier management and isolation.

Quick Start with Docker

# Build the Docker image
docker build -t newsarchiver .

# Run with default settings (archives stored in container)
docker run -p 5000:5000 newsarchiver

# Run with NAS storage mount
docker run -p 5000:5000 \
  -v /path/to/nas/backup:/data/archives \
  -e ARCHIVE_DIR=/data/archives \
  newsarchiver

Using Docker Compose

# Edit docker-compose.yml to configure your NAS mount path
vim docker-compose.yml

# Start the service
docker-compose up -d

# View logs
docker-compose logs -f

# Stop the service
docker-compose down

Configuration

The ARCHIVE_DIR environment variable controls where archived files are stored. To use NAS storage:

1. Edit docker-compose.yml and update the volume mount path
2. Set ARCHIVE_DIR to match the container path (e.g., /data/archives)
3. Restart the container

The archived data will persist even if the container is removed, as it's stored in a Docker volume or mounted NAS directory.

Stopping Services

To stop all NewsArchiver services:

./stop_services.sh