04f7365ac4
refactor: split into StockDocs (core), stockdocs-scraper, stockdocs-mcp
...
CI / lint (push) Waiting to run
CI / test (push) Waiting to run
CI / docker-build (push) Waiting to run
CI / security (push) Waiting to run
CI / build-result (push) Blocked by required conditions
Move the RSS scraper into its own repository and the MCP data
server into its own repository. This repo keeps the article
server, AI processor, and embedding service. Compose and pyproject
trimmed accordingly.
2026-08-21 18:33:59 +00:00
8030ef8efe
Merge pull request 'chore: remove dev artifacts, fix hardcoded path, add tests + license' ( #8 ) from improve/v1
CI / lint (push) Waiting to run
CI / test (push) Waiting to run
CI / docker-build (push) Waiting to run
CI / security (push) Waiting to run
CI / build-result (push) Blocked by required conditions
2026-08-20 21:39:25 +00:00
1271f0b21b
chore: remove dev artifacts, fix hardcoded path, add tests + license
...
- Remove agent/agent.md (dev-time agent context dumps), .DS_Store,
committed venv configs (pyvenv.cfg), 0-byte runtime cache
- Remove hardcoded /home/userpath from cron_scraper feed lookup
- Replace ad-hoc test_implementation.py with pytest tests/test_scraper_cache.py
- ruff clean (33 fixes: bare excepts, unused Config, whitespace)
- Root pyproject.toml (activates shared Gitea CI), MIT LICENSE, README Tests
2026-08-20 21:39:04 +00:00
ef51ef635c
Merge pull request 'fix: close #6 - find and build all Dockerfiles in subdirectories' ( #7 ) from fix/issue-6 into master
...
Reviewed-on: https://git.example.com/jarianc/StockDocs/pulls/7
2026-07-05 00:24:24 -05:00
930189ac4f
fix: close #6 - find and build all Dockerfiles in subdirectories
...
CI docker-build job only checked root Dockerfile, but all
Dockerfiles live in subdirs (scraper/, articleServer/, etc.).
Now uses find to locate all Dockerfile/dockerfile variants
and builds each in its directory context.
2026-07-05 05:23:49 +00:00
db7bbf5907
Merge pull request 'fix: close #1 - add bot-detection evasion for Reuters scraping' ( #5 ) from fix/issue-1 into master
...
Reviewed-on: https://git.example.com/jarianc/StockDocs/pulls/5
2026-07-05 00:23:13 -05:00
6b8a87f9ef
fix: close #1 - add bot-detection evasion for Reuters scraping
...
- Add rotating User-Agent pool (5 browser profiles) to all scrapers
- Fix Playwright: use browser context for UA instead of broken set_extra_http_headers
- Fix Selenium: set general.useragent.override preference
- Fix newspaper4k: pass browser_user_agent to bypass bot detection
- Add random delays (0.5-2s pre-fetch, 1-4s post-load) to mimic human behavior
- Switch Reuters RSS from Google News proxy to official Reuters agency feed
- Add legitimacy headers (Accept, Accept-Language, Connection) to Playwright
- Apply all fixes to both scraper.py and cron_scraper.py
2026-07-05 05:21:48 +00:00
e9785b3ea2
CI: remove --no-cache for docker layer caching
2026-07-05 02:57:54 +00:00
b41a43bf89
CI: add generalized workflow
2026-07-05 02:46:40 +00:00
0fa94d3ee3
feat: enhance logging for AI service error diagnosis
...
Added comprehensive logging to diagnose 'Expecting value: line 1 column 1 (char 0)' errors in AI service responses. The changes include detailed logging of AI requests/responses, cache operations, and article processing steps to better identify when the AI service returns empty or invalid responses.
2026-02-02 12:28:12 -06:00
271ede7845
Enhance AI processor error logging for better debugging
2026-02-02 10:56:10 -06:00
876cfddce7
Fix AI processor error handling for empty AI service responses
2026-02-02 10:52:34 -06:00
e29e5dbc76
Remove file extension filtering - all files in articles directory are valid articles
2026-02-02 10:38:35 -06:00
492b30a713
Fix cache manager to properly handle empty cache files and add diagnostic logging
2026-02-02 10:35:55 -06:00
35d8f44ee8
Fix cache logic bug that caused all files to be incorrectly marked as processed
2026-02-02 10:17:06 -06:00
218cad82cc
Fix cache file creation and validation issues - ensure cache file is properly created and handled
2026-02-02 10:02:59 -06:00
b8228d8e07
Fix pathing issue for article directory - use relative path ../scraper/articles
2026-02-02 09:55:24 -06:00
ad2ef2a87b
Remove cache file from version control, fix cache initialization logic
2026-02-02 09:50:42 -06:00
e9c66da456
Fix AI processor cache initialization issue and create empty cache file
2026-02-02 09:47:35 -06:00
6ce68a9037
Add key to api in docker
2026-02-02 09:37:06 -06:00
abb63db4b7
enhancement: Make article processor more robust with multiple path support and fallback file format detection
2026-02-02 09:31:14 -06:00
ba8f24fcbd
fix: Update scraper directory path for Docker container compatibility and add debugging
2026-02-02 09:27:02 -06:00
0b82bb13c4
fix: Handle missing cache_stats key in logging to prevent KeyError
2026-02-02 09:24:02 -06:00
e7b31a510a
fix: Complete removal of relative imports to resolve Docker container issues
2026-02-02 09:20:44 -06:00
5c47f6c235
fix: Resolve Docker import errors and relative import issues
2026-02-02 09:05:53 -06:00
e641a935a7
fix: Remove logging from requirements.txt due to Docker build error
2026-02-02 08:58:40 -06:00
714f5380e7
feat: Implement AI processor with fact extraction capabilities for articles from scraper
2026-02-02 08:53:58 -06:00
cb6c42fa65
gutting old ai_processor code that's not usful
2026-02-02 08:34:48 -06:00
9ab3e459e5
fix logging path in embedding pipe
2026-02-02 08:30:22 -06:00
68704b6edd
Add API key authentication for AI service connections
2026-02-02 04:29:50 -06:00
f6ded4fc14
Uncomment ai_processor service in docker-compose.yml to enable AI processing
2026-02-02 01:36:38 -06:00
dfa729f204
Fix embedding Dockerfile by removing unnecessary build dependencies
2026-02-02 01:28:26 -06:00
1dd9337c39
Fix embedding service by removing unused openai import and cleaning up dependencies
2026-02-02 01:23:14 -06:00
da03b3e847
Clean up requirements files and remove unnecessary dependencies including PyTorch
2026-02-01 21:11:22 -06:00
157e4541fc
Implement enhanced cache system with batch processing and two-phase processing approach
2026-02-01 21:03:00 -06:00
56231fa354
feat: Implement automated cron job setup for embedding pipeline and clean up AI processor fact extraction logic
...
• Created automated setup_embedding_cron_auto.sh script that fully configures cron jobs without manual intervention
• Enhanced embedding pipeline logging and error handling
• Simplified AI processor to focus on core fact extraction functionality
• Added proper logging to all scripts for better monitoring
2026-02-01 20:39:07 -06:00
6eb5e05b1e
get rid of cron.logs
2026-02-01 15:08:35 -06:00
8734ef9fcc
fix git ignore
2026-02-01 15:00:53 -06:00
211cf0b89e
added embedder to docker compose file and turned into cron job
2026-02-01 14:52:33 -06:00
91860211ad
Install dependencies and update requirements.txt for ChromaDB query functionality
2026-02-01 10:21:00 -06:00
c8a712d776
feat: implement enhanced data pipeline with Prometheus telemetry and Grafana integration
2026-01-31 23:56:06 -06:00
5e52142909
Add article server deployment to docker-compose and remove scraper service
...
Added article-server service to docker-compose.yml that builds from articleServer directory and maps articles volume from /home/user/StockDocs/scraper/articles. Removed the scraper service as it's no longer needed. The article server will now serve articles on port 5008.
2026-01-31 23:22:46 -06:00
c6f3cb3e65
Update n8n webhook URL to production endpoint
2026-01-31 20:52:53 -06:00
64e81869ea
Restore /facts endpoint and maintain all MCP Server functionality
2026-01-31 13:10:59 -06:00
ec756240b1
Implement all MCP Server endpoints from OpenAPI specification
2026-01-31 13:09:31 -06:00
1a62e2bd9b
cleaned up mcp server andn added /facts endpoint
2026-01-31 13:05:23 -06:00
0453faccb4
Add comprehensive agent.md overview file for repository projects
2026-01-31 12:36:08 -06:00
cc192cf566
Implement article filtering to avoid reprocessing already downloaded articles
2026-01-31 12:06:37 -06:00
1cbeb55463
Fix filename sanitization to resolve Linux permission denied errors
2026-01-31 11:53:32 -06:00
7808922aac
Remove U.S. News – Money RSS feed that was causing connection issues
2026-01-31 11:45:44 -06:00
