• Created automated setup_embedding_cron_auto.sh script that fully configures cron jobs without manual intervention
• Enhanced embedding pipeline logging and error handling
• Simplified AI processor to focus on core fact extraction functionality
• Added proper logging to all scripts for better monitoring
This fully implements the Stock Docs Project with full Docker Containerization support. This is a working prototype that is actively running on the Media Server. There's a few issues noted, including the following:
- Support for some sites could be improved. Reuters has many articles behind an adblock and some websites present banners that don't need to be processed by our AI engine
- Some caching could be smarter. As the size of files grows, it will get expensive to search through all files to be sure we've not scraped it, ai proccessed it or embedded it.
- Logging could be improved to be much better than just print statements and telemetry could be sent for dashboard monitoring if this were ever to become a full service where we cared about reliability.
- MCP server has been noted to return some poorly matching results. Would be better if it returned nothing at all. And should never really return banners or ads as that provides awful input for the model. Perhaps the model could be told to not care about this, but it's better to just never show irrelevant info to the model
I think this is an overall really good jumping off point, and we've already gotten to see the max capabilities of our system so far. It's a major win to have the Scraper for instance running at all times getting articles from across the web. I look forward to expending this scraper in the near future for projects like scraping all local news websites in the US or general scraping and monitoring of websites.
Co-authored-by: Jarian Cottingham <jariancottingham@dev-machine.local>
Co-authored-by: jarianc <user@example.com>
Reviewed-on: http://git.example.com/jarianc/StockDocs/pulls/2
