5 Commits

Author	SHA1	Message	Date
Jarian Cottingham	6b8a87f9ef	fix: close #1 - add bot-detection evasion for Reuters scraping ... - Add rotating User-Agent pool (5 browser profiles) to all scrapers - Fix Playwright: use browser context for UA instead of broken set_extra_http_headers - Fix Selenium: set general.useragent.override preference - Fix newspaper4k: pass browser_user_agent to bypass bot detection - Add random delays (0.5-2s pre-fetch, 1-4s post-load) to mimic human behavior - Switch Reuters RSS from Google News proxy to official Reuters agency feed - Add legitimacy headers (Accept, Accept-Language, Connection) to Playwright - Apply all fixes to both scraper.py and cron_scraper.py	2026-07-05 05:21:48 +00:00
Jarian Cottingham	7808922aac	Remove U.S. News – Money RSS feed that was causing connection issues	2026-01-31 11:45:44 -06:00
Jarian Cottingham	a07c637984	Remove problematic NASDAQ feed that was holding up threads	2026-01-31 11:24:59 -06:00
Jarian Cottingham	e207823427	Changing out AP rss feed	2025-10-17 01:59:22 -05:00
Jarian Cottingham	cd67adaae4	Initial Commit	2025-07-05 05:35:29 +00:00