# NewsArchiver

A news article archiving system that monitors RSS feeds and stores both raw HTML snapshots and parsed article content.

## Features

- Monitors 37 news websites via RSS feeds (business + technology)
- High-fidelity web page archiving using SingleFile CLI
- Content extraction with Trafilatura
- SQLite database for caching and duplicate detection
- Flask web interface for browsing archived content
- Background scheduler for automated daily archiving (via cron)

## Quick Start

```bash
# Install dependencies
pip install -r requirements.txt

# Setup cron job (recommended for production)
./setup_cron.sh

# Run archiver (one-time)
python run_archiver.py --run

# Start web interface
python run_archiver.py --serve
```

## Setup Scripts

- `setup_cron.sh` - Configure cron job for automated archiving every 15 minutes
- `stop_services.sh` - Stop all NewsArchiver services gracefully

## Cron Job Setup

For production use, run the setup script:

```bash
./setup_cron.sh
```

This will:
- Install dependencies if needed
- Start the web server
- Configure cron to run archiver every 15 minutes
- Log output to `/tmp/newsarchiver_cron.log`

The cron job runs independently of your terminal session, ensuring reliability.

## Project Structure

```
NewsArchiver/
├── run_archiver.py          # Main CLI entry point
├── scheduler.py             # Background scheduler (APScheduler)
├── rss_processor.py         # RSS feed fetching & caching
├── content_extractor.py     # HTML parsing & content extraction
├── storage_manager.py       # SQLite storage & file organization
├── archive_engine.py        # Orchestration engine
├── web_interface.py         # Flask web server
├── singlefile_archive.py    # SingleFile CLI integration
├── rss_feeds.json           # RSS feed configuration (37 feeds)
├── archival_data/           # Archive output directory
│   ├── cache.db            # SQLite cache database
│   ├── processing.log      # Processing logs
│   └── websites/           # Archived news sources
└── templates/               # Flask templates
```

## Usage

### Archive All Sources

```bash
python run_archiver.py --run
```

### Background Scheduler

**Recommended:** Use the setup script to configure cron:

```bash
./setup_cron.sh
```

**Manual approach** (not recommended - can be killed by terminal disconnect):

```bash
# Run every 15 minutes
python run_archiver.py --interval 15

# Run every hour
python run_archiver.py --interval 60
```

### Web Interface

```bash
python run_archiver.py --serve

# Visit http://localhost:5000
```

## Database

The SQLite database (`archival_data/cache.db`) stores:

- **articles**: 2378+ archived articles with metadata
- **article_archives**: Mapping of article URLs to archived HTML files
- **processing_log**: Processing history and errors

## Supported Sources

**Business/Finance (29):** Associated Press, BBC News, The Guardian, The Economist, Financial Times, Fortune, Barchart News, Investor's Business Daily, Financial Samurai, MoneyWeek, Finance Monthly, European Financial Review, World Finance, Fox Business, FinanceAsia, CNBC, Markets Insider, Economy Watch, CFI.co, MarketWatch, Wall Street Journal, Investing.com, International Business Times, Seeking Alpha, Motley Fool, TheStreet, MarketBeat, Money, Global Finance Magazine

**Technology (8):** 404 Media, Mac Rumors, The Verge, TechCrunch, WIRED, Hacker News, ZDNet, Engadget

**Total:** 37 sources (see `rss_feeds.json`)

## Requirements

- Python 3.8+
- Flask, Trafilatura, feedparser, APScheduler, requests, beautifulsoup4
- SingleFile CLI (optional, for web page archiving)

## Docker Deployment

The NewsArchiver can be deployed using Docker for easier management and isolation.

### Quick Start with Docker

```bash
# Build the Docker image
docker build -t newsarchiver .

# Run with default settings (archives stored in container)
docker run -p 5000:5000 newsarchiver

# Run with NAS storage mount
docker run -p 5000:5000 \
  -v /path/to/nas/backup:/data/archives \
  -e ARCHIVE_DIR=/data/archives \
  newsarchiver
```

### Using Docker Compose

```bash
# Edit docker-compose.yml to configure your NAS mount path
vim docker-compose.yml

# Start the service
docker-compose up -d

# View logs
docker-compose logs -f

# Stop the service
docker-compose down
```

### Configuration

The `ARCHIVE_DIR` environment variable controls where archived files are stored. To use NAS storage:

1. Edit `docker-compose.yml` and update the volume mount path
2. Set `ARCHIVE_DIR` to match the container path (e.g., `/data/archives`)
3. Restart the container

The archived data will persist even if the container is removed, as it's stored in a Docker volume or mounted NAS directory.

## Stopping Services

To stop all NewsArchiver services:

```bash
./stop_services.sh
```