8 Commits

Author	SHA1	Message	Date
Jarian Cottingham	1271f0b21b	chore: remove dev artifacts, fix hardcoded path, add tests + license ... - Remove agent/agent.md (dev-time agent context dumps), .DS_Store, committed venv configs (pyvenv.cfg), 0-byte runtime cache - Remove hardcoded /home/userpath from cron_scraper feed lookup - Replace ad-hoc test_implementation.py with pytest tests/test_scraper_cache.py - ruff clean (33 fixes: bare excepts, unused Config, whitespace) - Root pyproject.toml (activates shared Gitea CI), MIT LICENSE, README Tests	2026-08-20 21:39:04 +00:00
Jarian Cottingham	6b8a87f9ef	fix: close #1 - add bot-detection evasion for Reuters scraping ... - Add rotating User-Agent pool (5 browser profiles) to all scrapers - Fix Playwright: use browser context for UA instead of broken set_extra_http_headers - Fix Selenium: set general.useragent.override preference - Fix newspaper4k: pass browser_user_agent to bypass bot detection - Add random delays (0.5-2s pre-fetch, 1-4s post-load) to mimic human behavior - Switch Reuters RSS from Google News proxy to official Reuters agency feed - Add legitimacy headers (Accept, Accept-Language, Connection) to Playwright - Apply all fixes to both scraper.py and cron_scraper.py	2026-07-05 05:21:48 +00:00
Jarian Cottingham	c6f3cb3e65	Update n8n webhook URL to production endpoint	2026-01-31 20:52:53 -06:00
Jarian Cottingham	03b728d172	Improve logging in scraper with timestamps and focused new file notifications	2026-01-31 11:16:05 -06:00
Jarian Cottingham	936dd4dc28	Force exit if RSS feed file not found, as requested	2026-01-31 11:02:54 -06:00
Jarian Cottingham	874c64c9d1	Fix file path handling and add robust error handling for cron job execution	2026-01-31 11:00:14 -06:00
Jarian Cottingham	585499a171	Add webhook functionality to send new articles to external endpoint	2026-01-31 05:28:18 -06:00
Jarian Cottingham	b8a24a9fb1	feat(scraper): Add cron_scraper.py to fix freezing issues	2026-01-31 00:03:06 -06:00