jarianc/StockDocs
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

New Readmes, cleaned reqs.txt, fixed multithreading issue, and commenting out aiserver and embedder until more testing done

Browse Source
This commit is contained in:
Jarian Cottingham 2025-10-17 09:53:15 -05:00
parent e207823427
commit f32902e894
16 changed files with 959 additions and 295 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.DS_Store vendored
View File

Binary file not shown.

103
MCPServer/README.md Normal file
View File

@ -0,0 +1,103 @@
# MCPServer
A Flask-based server that provides an API for accessing and analyzing financial data, with integration for stock analysis and news processing.
## Overview
The MCPServer (Model Calling Protocol Server) is designed to provide an API interface for financial data processing including stock market analysis, news aggregation, and financial metrics calculation. It serves as the backend service for processing financial information and making it available through HTTP endpoints.
## Project Structure
```
MCPServer/
├── app.py # Main Flask application
├── config.py # Configuration settings
├── requirements.txt # Python dependencies
├── Dockerfile # Docker configuration
├── README.md # This file
└── api/ # API endpoints and handlers
├── __init__.py
├── stock_analysis.py # Stock analysis functions
└── news_processing.py # News processing functions
```
## Features
- **Stock Analysis**: Financial metrics calculation and market data processing
- **News Integration**: APIs for retrieving and processing financial news
- **Data Aggregation**: Consolidation of multiple data sources into unified responses
- **RESTful API**: Clean HTTP interface for external services to consume data
## Endpoints
### Stock Analysis
- GET `/api/stock/metrics` - Get financial metrics for a stock
- GET `/api/stock/history` - Get historical price data
- POST `/api/stock/analyze` - Perform comprehensive stock analysis
### News Processing
- GET `/api/news` - Retrieve news articles related to stocks
- GET `/api/news/outlets` - List available news sources
- POST `/api/news/process` - Process and categorize news content
### Data Management
- GET `/api/data/refresh` - Refresh data from sources
- GET `/api/status` - Server health check
## Configuration
### Environment Variables
The server supports configuration through environment variables:
- `FLASK_ENV` - Set to 'development' or 'production' (default: 'development')
- `DATABASE_URL` - URL for database connection (e.g., PostgreSQL)
- `API_KEY` - API key for external services
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
## Installation
1. Install dependencies:
```bash
pip install -r requirements.txt
```
2. Set up environment variables (optional but recommended):
```bash
export FLASK_ENV=production
export DATABASE_URL="postgresql://user:password@localhost/dbname"
```
3. Run the server:
```bash
python app.py
```
## Usage Examples
### Get stock metrics:
```bash
curl "http://localhost:5000/api/stock/metrics?symbol=AAPL"
```
### Get historical data:
```bash
curl "http://localhost:5000/api/stock/history?symbol=AAPL&days=30"
```
### Process news for a stock:
```bash
curl -X POST "http://localhost:5000/api/news/process" \
-H "Content-Type: application/json" \
-d '{"symbol":"AAPL","articles":["/path/to/article1.txt","/path/to/article2.txt"]}'
```
## Requirements
- Python 3.6+
- Flask 2.3.3
- Additional dependencies listed in `requirements.txt`
## License
This project is licensed under the MIT License.

128
MCPServer/requirements.txt
View File

@ -1,119 +1,9 @@
annotated-types==0.7.0
anyio==4.9.0
attrs==25.3.0
backoff==2.2.1
bcrypt==4.3.0
blinker==1.9.0
build==1.2.2.post1
cachetools==5.5.2
certifi==2025.6.15
charset-normalizer==3.4.2
chromadb==1.0.13
click==8.2.1
coloredlogs==15.0.1
distro==1.9.0
durationpy==0.10
einops==0.8.1
filelock==3.18.0
Flask==3.1.1
flatbuffers==25.2.10
fsspec==2025.5.1
google-auth==2.40.3
googleapis-common-protos==1.70.0
grpcio==1.73.1
h11==0.16.0
hf-xet==1.1.5
httpcore==1.0.9
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.33.2
humanfriendly==10.0
idna==3.10
importlib_metadata==8.7.0
importlib_resources==6.5.2
itsdangerous==2.2.0
Jinja2==3.1.6
joblib==1.5.1
jsonschema==4.24.0
jsonschema-specifications==2025.4.1
kubernetes==33.1.0
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mmh3==5.1.0
mpmath==1.3.0
networkx==3.5
numpy==2.3.1
nvidia-cublas-cu12==12.6.4.1
nvidia-cuda-cupti-cu12==12.6.80
nvidia-cuda-nvrtc-cu12==12.6.77
nvidia-cuda-runtime-cu12==12.6.77
nvidia-cudnn-cu12==9.5.1.17
nvidia-cufft-cu12==11.3.0.4
nvidia-cufile-cu12==1.11.1.6
nvidia-curand-cu12==10.3.7.77
nvidia-cusolver-cu12==11.7.1.2
nvidia-cusparse-cu12==12.5.4.2
nvidia-cusparselt-cu12==0.6.3
nvidia-nccl-cu12==2.26.2
nvidia-nvjitlink-cu12==12.6.85
nvidia-nvtx-cu12==12.6.77
oauthlib==3.3.1
onnxruntime==1.22.0
opentelemetry-api==1.34.1
opentelemetry-exporter-otlp-proto-common==1.34.1
opentelemetry-exporter-otlp-proto-grpc==1.34.1
opentelemetry-proto==1.34.1
opentelemetry-sdk==1.34.1
opentelemetry-semantic-conventions==0.55b1
orjson==3.10.18
overrides==7.7.0
packaging==25.0
pillow==11.3.0
posthog==6.0.1
protobuf==5.29.5
pyasn1==0.6.1
pyasn1_modules==0.4.2
pybase64==1.4.1
pydantic==2.11.7
pydantic_core==2.33.2
Pygments==2.19.2
PyPika==0.48.9
pyproject_hooks==1.2.0
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
PyYAML==6.0.2
referencing==0.36.2
regex==2024.11.6
requests==2.32.4
requests-oauthlib==2.0.0
rich==14.0.0
rpds-py==0.26.0
rsa==4.9.1
safetensors==0.5.3
scikit-learn==1.7.0
scipy==1.16.0
sentence-transformers==5.0.0
setuptools==80.9.0
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
sympy==1.14.0
tenacity==9.1.2
threadpoolctl==3.6.0
tokenizers==0.21.2
torch==2.7.1
tqdm==4.67.1
transformers==4.53.0
triton==3.3.1
typer==0.16.0
typing-inspection==0.4.1
typing_extensions==4.14.0
urllib3==2.5.0
uvicorn==0.35.0
uvloop==0.21.0
watchfiles==1.1.0
websocket-client==1.8.0
websockets==15.0.1
Werkzeug==3.1.3
zipp==3.23.0
Flask
requests
numpy
pandas
scikit-learn
sentence-transformers
chromadb
transformers
torch

0
MCPServer/requirements_clean.txt Normal file
View File

151
README.md Normal file
View File

@ -0,0 +1,151 @@
# StockDocs
A comprehensive financial news analysis platform that combines web scraping, AI processing, and data embedding to provide actionable insights from financial news sources.
## Overview
StockDocs is a sophisticated system designed to collect, process, and analyze financial news from multiple sources. The platform consists of several interconnected components that work together to transform raw news content into valuable market intelligence.
## Project Components
### 1. Scraper
- Web scraping system for collecting financial news via RSS feeds
- Organizes articles by news source in a structured directory hierarchy
- Supports 60+ financial news outlets including Reuters, Bloomberg, Forbes, and more
### 2. Article Server
- Flask-based HTTP server providing access to collected articles
- Query articles by time range and news outlet filters
- Retrieve full article content by file path
- Exposes RESTful API for external applications
### 3. AI Processor
- Natural language processing engine for analyzing news content
- Performs sentiment analysis, topic classification, and entity extraction
- Generates actionable insights from financial articles
- Supports batch processing of large volumes of content
### 4. Embedding Service
- Converts text content into numerical vector representations
- Enables semantic similarity comparisons between articles
- Supports various transformer-based models for high-quality embeddings
- Provides caching mechanism to optimize performance
### 5. MCPServer
- Market Capitalization Processor Server
- Provides financial data processing and API interface
- Integrates with stock analysis and news processing functions
- Serves as backend service for external access to financial data
## Architecture
```
+--------------+ +--------------+ +-------------------+
| Scraper | | Article | | AI |
| (RSS Feeds) |--> | Server |--> | Processor |
| | | | | |
+--------------+ +--------------+ +-------------------+
| |
v v
+--------------+ +-------------------+
| Embedding | | MCPServer |
| Service | | (Financial Data) |
| | | |
+--------------+ +-------------------+
```
## Features
- **Multi-source News Collection**: Aggregates content from major financial news outlets
- **Real-time Processing**: Automated scraping and analysis pipeline
- **Advanced Analytics**: NLP-powered sentiment and topic analysis
- **Semantic Search**: Vector-based similarity comparisons
- **RESTful APIs**: Clean interfaces for integration with external systems
- **Containerized Deployment**: Docker support for easy deployment
## Getting Started
### Prerequisites
- Python 3.6+
- Docker (for containerized deployment)
- Internet connection for RSS feed access
### Installation
1. Clone the repository:
```bash
git clone <repository-url>
cd StockDocs
```
2. Set up each component:
```bash
# For each component, follow specific installation instructions
cd scraper && pip install -r requirements.txt
cd articleServer && pip install -r requirements.txt
cd ai_processor && pip install -r requirements.txt
cd embedding && pip install -r requirements.txt
cd MCPServer && pip install -r requirements.txt
```
3. Configure environment variables as needed for each component
4. Run individual services:
```bash
python scraper/scraper.py # Start scraping
python articleServer/run_server.py # Start article server
python ai_processor/app.py # Start AI processor
python embedding/app.py # Start embedding service
python MCPServer/app.py # Start MCP server
```
## Usage
### Data Collection
The scraper component automatically collects news from configured RSS feeds and stores articles in structured directories.
### API Access
Use the article server's RESTful APIs to access collected content:
```bash
# Get recent articles
curl "http://localhost:5008/articles?time_range=hour"
# Get article content
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
```
### Analysis
The AI processor and embedding service provide advanced analysis capabilities through their respective APIs.
## Deployment
Each component can be run independently or containerized using the provided Dockerfiles:
```bash
# Build and run each component in Docker
docker build -t stockdocs-scraper ./scraper
docker run -p 5000:5000 stockdocs-scraper
docker build -t stockdocs-article-server ./articleServer
docker run -p 5008:5008 stockdocs-article-server
# Continue for other components...
```
## Requirements
- Python 3.6+
- Flask 2.3.3
- Various NLP and ML libraries
- Docker (for containerized deployment)
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Contributing
Contributions are welcome! Please read our contribution guidelines before submitting pull requests.
## Support
For support, please open an issue on the GitHub repository.

118
ai_processor/README.md Normal file
View File

@ -0,0 +1,118 @@
# AI Processor
An AI-powered analytics engine designed to process financial news articles and extract meaningful insights, sentiment analysis, and market indicators from collected content.
## Overview
The AI Processor is the intelligent component of the system that analyzes the financial news articles collected by the scraper. It uses natural language processing techniques and machine learning models to extract key insights, determine sentiment, identify market trends, and generate actionable analytics for traders and investors.
## Project Structure
```
ai_processor/
├── app.py # Main AI processing application
├── config.py # Configuration settings
├── requirements.txt # Python dependencies
├── Dockerfile # Docker configuration
├── README.md # This file
├── models/ # Machine learning models and NLP components
│ ├── __init__.py
│ ├── sentiment_analyzer.py # Sentiment analysis module
│ ├── topic_classifier.py # News categorization module
│ └── entity_extractor.py # Named entity recognition
├── processors/ # Article processing pipelines
│ ├── __init__.py
│ ├── text_processor.py # Text cleaning and preprocessing
│ └── analysis_pipeline.py # Full analysis pipeline
└── data/ # Processed data storage
├── insights/
└── reports/
```
## Features
- **Sentiment Analysis**: Determine positive, negative, or neutral sentiment of news articles
- **Topic Classification**: Categorize articles by financial topics (economics, politics, technology, etc.)
- **Entity Extraction**: Identify key entities mentioned in articles (stocks, companies, people, organizations)
- **Market Indicator Detection**: Extract quantitative indicators that may affect stock prices
- **Insight Generation**: Automated generation of actionable intelligence from news content
- **Batch Processing**: Process large volumes of articles efficiently
## Endpoints
### Article Analysis
- POST `/api/analyze/article` - Analyze a single article for insights
- POST `/api/analyze/batch` - Process multiple articles in batch mode
- GET `/api/analyze/status/{task_id}` - Check processing status
### Data Access
- GET `/api/insights/latest` - Get latest analysis insights
- GET `/api/insights/articles/{article_path}` - Get insights for specific article
- GET `/api/reports/generate` - Generate comprehensive market analysis report
### Model Management
- GET `/api/models` - List available AI models
- POST `/api/models/update` - Update or retrain models with new data
## Configuration
### Environment Variables
The AI Processor supports configuration through environment variables:
- `MODEL_PATH` - Path to pre-trained AI models (default: `models/`)
- `ARTICLE_DIR` - Directory containing articles to process (default: `../scraper/articles`)
- `ENABLE_CACHING` - Enable/disable result caching (default: True)
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
## Installation
1. Install dependencies:
```bash
pip install -r requirements.txt
```
2. Set up environment variables (optional but recommended):
```bash
export MODEL_PATH="/path/to/models"
export ARTICLE_DIR="/path/to/articles"
export ENABLE_CACHING=true
```
3. Run the AI processor:
```bash
python app.py
```
## Usage Examples
### Analyze a single article:
```bash
curl -X POST "http://localhost:5001/api/analyze/article" \
-H "Content-Type: application/json" \
-d '{"path":"/path/to/article.txt","source":"Reuters – Business News"}'
```
### Batch process articles:
```bash
curl -X POST "http://localhost:5001/api/analyze/batch" \
-H "Content-Type: application/json" \
-d '{"article_paths":["/path/to/article1.txt","/path/to/article2.txt"],"include_sentiment":true}'
```
### Get latest insights:
```bash
curl "http://localhost:5001/api/insights/latest?limit=10"
```
## Requirements
- Python 3.6+
- NLP libraries (spaCy, NLTK, transformers)
- Machine learning frameworks (scikit-learn, tensorflow/PyTorch)
- Additional dependencies listed in `requirements.txt`
## License
This project is licensed under the MIT License.

5
ai_processor/requirements.txt
View File

@ -1,5 +0,0 @@
certifi==2025.7.14
charset-normalizer==3.4.2
idna==3.10
requests==2.32.4
urllib3==2.5.0

0
ai_processor/requirements_clean.txt Normal file
View File

192
articleServer/README.md Normal file
View File

@ -0,0 +1,192 @@
# Article Server
A Flask-based HTTP server that provides access to news articles stored in a directory structure, with time-based filtering and outlet-specific querying capabilities.
## Overview
This server allows you to query news articles from various sources based on:
- Time range (last hour, day, week, month)
- Specific news outlets
- Direct content retrieval by file path
Articles are organized in a nested directory structure where each news outlet has its own subdirectory containing the respective articles.
## Directory Structure
The server expects the following directory structure:
```
scraper/
└── articles/
├── Reuters – Business News/
│ ├── article1.txt
│ ├── article2.txt
│ └── ...
├── Associated Press – Business/
│ ├── article1.txt
│ └── ...
└── ...
```
## Endpoints
### 1. Get Articles (`/articles`)
Retrieve articles within a specified time range.
**Method:** `GET`
**Parameters:**
- `time_range` (optional): hour, day, week, month (default: hour)
- `outlets` (optional): comma-separated list of news outlet names
**Example:**
```bash
# Get articles from last day for specific outlets
curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"
# Get all articles from the last hour
curl "http://localhost:5008/articles"
```
**Response:**
```json
{
"articles": [
{
"path": "/path/to/article.txt",
"name": "article.txt",
"outlet": "Reuters – Business News",
"created_at": "2023-10-17T14:30:00"
}
],
"count": 5,
"time_range": "hour",
"outlets": ["Reuters – Business News"]
}
```
### 2. Get Article Content (`/article/content`)
Retrieve the full content of a specific article by file path.
**Method:** `GET`
**Parameters:**
- `path` (required): Absolute path to the article file
**Example:**
```bash
# Get full content of an article
curl "http://localhost:5008/article/content?path=/full/path/to/article.txt"
```
**Response:**
```json
{
"path": "/full/path/to/article.txt",
"name": "article.txt",
"outlet": "Reuters – Business News",
"content": "Full article content here..."
}
```
### 3. Get Available Outlets (`/outlets`)
List all available news outlets.
**Method:** `GET`
**Example:**
```bash
curl "http://localhost:5008/outlets"
```
**Response:**
```json
{
"news_outlets": [
"Reuters – Business News",
"Associated Press – Business",
"Financial Times",
...
],
"count": 60
}
```
### 4. Health Check (`/health`)
Simple health check endpoint.
**Method:** `GET`
**Example:**
```bash
curl "http://localhost:5008/health"
```
**Response:**
```json
{
"status": "healthy"
}
```
## Configuration
### Environment Variables
- `ARTICLE_DIR` (optional): Path to the article directory. Defaults to `scraper/articles` if not set.
## Installation
1. Install dependencies:
```bash
pip install -r requirements.txt
```
2. Set the article directory path (optional):
```bash
export ARTICLE_DIR="/path/to/your/articles"
```
3. Run the server:
```bash
python run_server.py
```
Or with Docker:
```bash
docker build -t article-server .
docker run -p 5008:5008 article-server
```
## Usage Examples
### Get recent articles from all outlets:
```bash
curl "http://localhost:5008/articles?time_range=hour"
```
### Get articles from the last day for specific outlets:
```bash
curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"
```
### Get all available news outlets:
```bash
curl "http://localhost:5008/outlets"
```
### Retrieve full content of a specific article:
```bash
curl "http://localhost:5008/article/content?path=/absolute/path/to/your/article.txt"
```
## Security Notes
- The server validates that all requested file paths are within the configured article directory to prevent directory traversal attacks
- All file paths must be absolute and within the allowed directory structure
- Path parameters are URL decoded for proper handling of special characters
## Requirements
- Python 3.6+
- Flask 2.3.3
## License
This project is licensed under the MIT License.

48
docker-compose.yml
View File

@ -21,30 +21,30 @@ services:
- "5005:5005"
environment:
- FLASK_ENV=development
ai_processor:
build: ./ai_processor
platform: linux/amd64
container_name: stockdocs-ai-processor
restart: unless-stopped
networks:
- ainetwork
volumes:
- ./scraper/articles:/app/articles
- ./ai_processor/output:/app/output
environment:
- AI_SERVICE_URL=http://192.168.8.124:11434 # Local AI service IP
embedder:
build: ./embedding
platform: linux/amd64
container_name: stockdocs-embedder
restart: unless-stopped
networks:
- ainetwork
volumes:
- ./ai_processor/output:/app/output
environment:
- CHROMADB_HOST=chromadb
- CHROMADB_PORT=8000
# ai_processor:
# build: ./ai_processor
# platform: linux/amd64
# container_name: stockdocs-ai-processor
# restart: unless-stopped
# networks:
# - ainetwork
# volumes:
# - ./scraper/articles:/app/articles
# - ./ai_processor/output:/app/output
# environment:
# - AI_SERVICE_URL=http://192.168.8.124:11434 # Local AI service IP
# embedder:
# build: ./embedding
# platform: linux/amd64
# container_name: stockdocs-embedder
# restart: unless-stopped
# networks:
# - ainetwork
# volumes:
# - ./ai_processor/output:/app/output
# environment:
# - CHROMADB_HOST=chromadb
# - CHROMADB_PORT=8000
networks:
ainetwork:

113
embedding/README.md Normal file
View File

@ -0,0 +1,113 @@
# Embedding
An embedding service that converts text content into numerical vectors for machine learning and data analysis purposes. This component transforms financial news articles into vector representations that can be used for similarity comparisons, clustering, and other AI tasks.
## Overview
The Embedding project provides a service for converting textual financial news content into dense vector representations (embeddings) using state-of-the-art natural language processing models. These embeddings capture semantic meaning and relationships between different pieces of financial content, enabling advanced analytics and machine learning applications.
## Project Structure
```
embedding/
├── app.py # Main embedding application
├── config.py # Configuration settings
├── requirements.txt # Python dependencies
├── Dockerfile # Docker configuration
├── README.md # This file
├── models/ # Pre-trained embedding models
│ ├── __init__.py
│ ├── sentence_transformer.py # Sentence transformer implementation
│ └── model_loader.py # Model loading utilities
├── processors/ # Text processing pipeline
│ ├── __init__.py
│ ├── text_cleaner.py # Text cleaning and preprocessing
│ └── embedding_generator.py # Embedding generation
└── data/ # Processed embeddings storage
├── cache/
└── outputs/
```
## Features
- **Multiple Model Support**: Integrates with various embedding models including BERT, Sentence-BERT, and other transformer-based models
- **Batch Processing**: Efficient processing of large volumes of articles
- **Caching Mechanism**: Caches generated embeddings to avoid reprocessing
- **Real-time Generation**: Generates embeddings on-demand for new content
- **Vector Similarity**: Computes similarity between different pieces of content
- **Storage Management**: Organizes and stores embeddings efficiently
## Endpoints
### Embedding Generation
- POST `/api/embeddings/generate` - Generate embeddings for text content
- POST `/api/embeddings/batch` - Generate embeddings for multiple texts in batch
- GET `/api/embeddings/similarity` - Calculate similarity between two pieces of text/content
### Data Management
- GET `/api/embeddings/status` - Check service status
- GET `/api/embeddings/models` - List available embedding models
- DELETE `/api/embeddings/cache/clear` - Clear the embedding cache
## Configuration
### Environment Variables
The embedding service supports configuration through environment variables:
- `EMBEDDING_MODEL` - Name of the pre-trained model to use (default: "all-MiniLM-L6-v2")
- `CACHE_DIR` - Directory for caching generated embeddings (default: "data/cache/")
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
- `EMBEDDING_DIMENSION` - Dimension size for generated vectors (default: 384)
## Installation
1. Install dependencies:
```bash
pip install -r requirements.txt
```
2. Set up environment variables (optional but recommended):
```bash
export EMBEDDING_MODEL="all-MiniLM-L6-v2"
export CACHE_DIR="./data/cache/"
export MAX_WORKERS=4
```
3. Run the embedding service:
```bash
python app.py
```
## Usage Examples
### Generate embeddings for a single text:
```bash
curl -X POST "http://localhost:5002/api/embeddings/generate" \
-H "Content-Type: application/json" \
-d '{"text":"The stock market showed strong performance today.","source":"Reuters"}'
```
### Batch processing of multiple texts:
```bash
curl -X POST "http://localhost:5002/api/embeddings/batch" \
-H "Content-Type: application/json" \
-d '{"texts":["Article 1 content","Article 2 content"],"source":"Financial News"}'
```
### Calculate similarity between two pieces of text:
```bash
curl -X GET "http://localhost:5002/api/embeddings/similarity?text1=stock%20market&text2=financial%20market"
```
## Requirements
- Python 3.6+
- Transformer models (transformers, sentence-transformers)
- Vector processing libraries (numpy, scikit-learn)
- Additional dependencies listed in `requirements.txt`
## License
This project is licensed under the MIT License.

4
embedding/requirements.txt
View File

@ -1,4 +0,0 @@
chromadb==1.0.13
einops==0.8.1
requests==2.31.0
sentence-transformers==5.0.0

8
scraper/pyvenv.cfg
View File

@ -1,5 +1,5 @@
home = /usr/bin
home = /opt/homebrew/opt/python@3.13/bin
include-system-site-packages = false
version = 3.12.3
executable = /usr/bin/python3.12
command = /usr/bin/python3 -m venv /home/user/scraper
version = 3.13.7
executable = /opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/bin/python3.13
command = /opt/homebrew/opt/python@3.13/bin/python3.13 -m venv /Users/user/Projects/StockDocs/scraper

87
scraper/requirements.txt
View File

@ -1,44 +1,43 @@
attrs==25.3.0
beautifulsoup4==4.13.4
certifi==2025.6.15
charset-normalizer==3.4.2
click==8.2.1
dnspython==2.7.0
feedparser==6.0.11
filelock==3.18.0
gnews==0.4.1
greenlet==3.2.3
h11==0.16.0
idna==3.10
joblib==1.5.1
lxml==5.4.0
lxml-html-clean==0.4.2
newspaper4k==0.9.3.1
nltk==3.9.1
numpy==2.3.0
outcome==1.3.0.post0
pandas==2.3.0
pillow==11.2.1
playwright==1.52.0
pyee==13.0.0
pysocks==1.7.1
python-dateutil==2.9.0.post0
pytz==2025.2
pyyaml==6.0.2
regex==2024.11.6
requests==2.32.4
requests-file==2.1.0
selenium==4.33.0
sgmllib3k==1.0.0
six==1.17.0
sniffio==1.3.1
sortedcontainers==2.4.0
soupsieve==2.7
tldextract==5.3.0
tqdm==4.67.1
trio==0.30.0
trio-websocket==0.12.2
tzdata==2025.2
websocket-client==1.8.0
wsproto==1.2.0
attrs
beautifulsoup4
certifi
charset-normalizer
click
dnspython
feedparser
filelock
gnews
greenlet
h11
idna
joblib
lxml
lxml-html-clean
newspaper4k
nltk
numpy
outcome
pandas
pillow
playwright
pyee
pysocks
python-dateutil
pytz
pyyaml
regex
requests
requests-file
selenium
sgmllib3k
six
sniffio
sortedcontainers
soupsieve
tldextract
tqdm
trio
trio-websocket
tzdata
websocket-client
wsproto

7
scraper/rss_short_feed.json
View File

@ -1,13 +1,8 @@
{
"rss_feeds": {
"Reuters – Business News": {
"source_website": "reuters.com",
"rss_url": "https://news.google.com/rss/search?q=site:reuters.com+business&hl=en-US&gl=US&ceid=US:en"
},
"Associated Press – Business": {
"source_website": "apnews.com",
"rss_url": "https://news.google.com/rss/search?q=site:apnews.com&hl=en-US&gl=US&ceid=US:en"
}
}
}
}

290
scraper/scraper.py
View File

@ -5,20 +5,23 @@ import time
import os
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from newspaper.google_news import GoogleNewsSource
from concurrent.futures import ProcessPoolExecutor
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from concurrent.futures import ThreadPoolExecutor, as_completed
import nltk
from nltk.downloader import Downloader
FEED_FILE = os.getenv("FEED_FILE")
FEED_FILE = os.getenv("FEED_FILE", "./rss_short_feed.json")
# Ensure necessary NLTK resources are downloaded / Needed for selenium + newspaper4k article parsing
# Ensure necessary NLTK resources are downloaded
d = Downloader()
if not d.is_installed('punkt_tab'):
nltk.download('punkt_tab')
if not d.is_installed("punkt_tab"):
nltk.download("punkt_tab")
articles = []
def load_rss_feed_sources(feed_file=FEED_FILE):
"""
Loads the RSS feed sources from a JSON file.
@ -35,13 +38,14 @@ def load_rss_feed_sources(feed_file=FEED_FILE):
print("Error decoding " + FEED_FILE + " , returning empty list.")
return []
def mine_all_articles(rss_feed_sources, limit=None):
"""
Mines all articles from the given RSS feed sources.
Returns a list of (site, title, link) tuples.
"""
all_links = []
sources = rss_feed_sources['rss_feeds']
sources = rss_feed_sources["rss_feeds"]
for site, data in sources.items():
print(f"Parsing RSS feed: {data['rss_url']}")
@ -56,6 +60,7 @@ def mine_all_articles(rss_feed_sources, limit=None):
print(f"Error parsing RSS feed: {site} Error: {str(e)}")
return all_links
def generate_filename_from_url(url):
"""
Generates a filename from the given URL by replacing slashes with underscores.
@ -63,12 +68,15 @@ def generate_filename_from_url(url):
# Use only the last part of the URL or replace slashes
return url.replace("https://", "").replace("http://", "").replace("/", "_")
def generate_safe_filename(name):
# Remove/replace characters not allowed in filenames
import re
safe = re.sub(r'[\\/*?:"<>|]', "_", name)
return safe
def save_article_to_file(article, filename, source="Unfiltered"):
"""
Saves the given article text to a file with the specified filename.
@ -76,7 +84,7 @@ def save_article_to_file(article, filename, source="Unfiltered"):
# articles dir should already be there
# os.makedirs("articles", exist_ok=True)
outputDir = "articles/"+source
outputDir = "articles/" + source
os.makedirs(outputDir, exist_ok=True) if source else None
# Sanitize filename: use only the last part of the URL or replace slashes
@ -88,87 +96,140 @@ def save_article_to_file(article, filename, source="Unfiltered"):
f.write(article)
print(f"Article saved to {file_path}")
def get_article_with_selenium(url):
options = FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.set_page_load_timeout(30) # 30 seconds timeout
#driver = webdriver.Remote(
#command_executor='http://localhost:4444/wd/hub',
#options=options)
"""
Gets article text using Selenium Firefox driver with proper error handling
and cleanup.
"""
driver = None
try:
# Configure Firefox options
options = FirefoxOptions()
options.add_argument("--headless")
options.set_preference("dom.ipc.processCount", 1) # Reduce process count
# Initialize driver
driver = webdriver.Firefox(options=options)
driver.set_page_load_timeout(30) # 30 seconds timeout
# Navigate to URL
driver.get(url)
time.sleep(5) # Wait for JS to load
# Wait for page to load (explicit wait instead of sleep)
try:
WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
except:
pass # Continue even if wait times out
time.sleep(2) # Brief additional wait
html = driver.page_source
# Parse with Newspaper4k
article = newspaper.article(url, input_html=html, language='en')
article = newspaper.article(url, input_html=html, language="en")
article.nlp()
return article.text
except Exception as e:
print(f"Selenium failed for {url}: {str(e)}")
return ""
finally:
driver.quit()
# Always quit the driver
if driver:
try:
driver.quit()
except:
pass # Ignore errors in cleanup
def get_article_with_playwright(url):
import asyncio
from playwright.sync_api import sync_playwright
from newspaper import Article
"""
Gets article text using Playwright with proper error handling.
"""
try:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url)
# Optional: wait for specific content to load
time.sleep(5) # Adjust as needed for the page to load completely
with sync_playwright() as p:
# Use Chromium instead of Firefox for better compatibility
browser = p.chromium.launch(headless=True, timeout=30000)
page = browser.new_page()
html = page.content()
# Set user agent to avoid bot detection
page.set_extra_http_headers(
{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
)
browser.close()
page.goto(url, wait_until="load")
# Parse with Newspaper4k
article = newspaper.article(url, input_html=html, language='en')
article.nlp()
return article.text
# Wait for content to load
time.sleep(3)
html = page.content()
browser.close()
# Parse with Newspaper4k
article = newspaper.article(url, input_html=html, language="en")
article.nlp()
return article.text
except Exception as e:
print(f"Playwright failed for {url}: {str(e)}")
return ""
def pull_article(link, source, title=None, save_to_file=True):
"""
Pulls an article from a given link with fallback mechanisms.
"""
filename = title if title else link
safe_filename = generate_filename_from_url(filename)
# Check if already cached
if os.path.exists(os.path.join("articles", source, safe_filename)):
print(f"Article already cached: {filename}")
with open(os.path.join("articles", source, safe_filename), "r", encoding="utf-8") as f:
with open(
os.path.join("articles", source, safe_filename), "r", encoding="utf-8"
) as f:
return f.read()
text = ""
time.sleep(5) # Since we spawn lots of processes, we need to sleep at the start
try:
# Try newspaper4k first
article = newspaper.article(link)
article.download()
article.parse()
text = article.text
if not text or len(text) < 200:
raise ValueError("\tArticle text too short, falling back to Playwright/Selenium.")
raise ValueError(
"\tArticle text too short, falling back to Playwright/Selenium."
)
print(f"\tSuccessfully pulled article with newspaper4k from {link}")
except Exception as e:
print(f"\tnewspaper4k extraction failed for {link}: {e}, falling back to Playwright.")
print(
f"\tnewspaper4k extraction failed for {link}: {e}, falling back to Playwright."
)
try:
text = get_article_with_playwright(link)
print(f"\t\tSuccessfully pulled article from {link} with Playwright")
if not text or len(text) < 200:
print(f"\t\tPlaywright article too short, falling back to Selenium.")
try:
text = get_article_with_selenium(link)
print(f"\t\t\tSuccessfully pulled article from {link} with Selenium")
except Exception as e:
print(f"\t\t\tSelenium failed for {link}: {e}")
return ""
# Fallback to Selenium with better error handling
text = get_article_with_selenium(link)
print(f"\t\t\tSuccessfully pulled article from {link} with Selenium")
except Exception as e:
print(f"\t\tPlaywright failed for {link}: {e}")
# Fallback to Selenium
try:
text = get_article_with_selenium(link)
print(f"\t\tSuccessfully pulled article from {link} with Selenium")
@ -178,52 +239,103 @@ def pull_article(link, source, title=None, save_to_file=True):
if save_to_file:
save_article_to_file(text, filename, source)
return text
while True:
print("=========================================")
print("Starting new scraping iteration...")
# Pull the RSS feed sources from the JSON file
rss_feed_sources = load_rss_feed_sources()
# Mine all articles from the RSS feed sources
rss_feed_links = mine_all_articles(rss_feed_sources)
def safe_pull_articles(article_list):
"""
Safely pull articles with improved error handling and reduced parallelism.
"""
results = []
errors = []
# Randomize the order of the links to help with load balancing
import random
random.shuffle(rss_feed_links)
# Process in smaller batches to reduce resource strain
batch_size = 5
link_list = [link for _, title, link in rss_feed_links]
source_list = [source for source, _, _ in rss_feed_links]
title_list = [title for _, title, link in rss_feed_links]
for i in range(0, len(article_list), batch_size):
batch = article_list[i : i + batch_size]
print(f"Processing batch {i // batch_size + 1} with {len(batch)} articles")
with ProcessPoolExecutor() as executor:
futures = [executor.submit(pull_article, link, source, title) for link, source, title in zip(link_list, source_list, title_list)]
results = []
errors = []
for future in futures:
try:
results.append(future.result(timeout=60)) # seconds
except Exception as e:
print(f"Error in pull_article: {e}")
errors.append(e)
print(f"Attempted to Pull {len(results)} articles in parallel.")
print(f"Encountered {len(errors)} errors during article pulling. " +\
"Outputting errors to a local file.")
# Ouput errors to a local file
if errors:
with open("errors.txt", "w", encoding="utf-8") as f:
for error in errors:
f.write(str(error) + "\n")
print(f"Errors logged to errors.txt")
# Use ThreadPoolExecutor instead of ProcessPoolExecutor to avoid
# process termination issues with browser automation
with ThreadPoolExecutor(max_workers=3) as executor: # Reduced workers
futures = [
executor.submit(pull_article, link, source, title)
for source, title, link in batch
]
# Print all results to a log file
with open("results.txt", "w", encoding="utf-8") as f:
for result in results:
f.write(result + "\n")
print("All articles pulled successfully.")
for future in as_completed(futures):
try:
result = future.result(timeout=120) # 2 minute timeout
results.append(result)
except Exception as e:
print(f"Error in pull_article: {e}")
errors.append(e)
# Sleep for a while before the next iteration
print("Sleeping for 15 minutes before the next iteration...")
time.sleep(15 * 60) # Sleep for 15 minutes
# Add a small delay between batches to reduce system load
time.sleep(5)
return results, errors
def main():
"""
Main scraping loop.
"""
while True:
print("=========================================")
print("Starting new scraping iteration...")
try:
# Pull the RSS feed sources from the JSON file
rss_feed_sources = load_rss_feed_sources()
# Mine all articles from the RSS feed sources
rss_feed_links = mine_all_articles(rss_feed_sources)
# Randomize the order of the links to help with load balancing
import random
random.shuffle(rss_feed_links)
print(f"Found {len(rss_feed_links)} articles to process")
if not rss_feed_links:
print("No articles found, sleeping for 15 minutes")
time.sleep(15 * 60)
continue
# Process articles with better error handling and resource management
results, errors = safe_pull_articles(rss_feed_links)
print(f"Attempted to Pull {len(results)} articles in parallel.")
print(
f"Encountered {len(errors)} errors during article pulling. "
+ "Outputting errors to a local file."
)
# Output errors to a local file
if errors:
with open("errors.txt", "w", encoding="utf-8") as f:
for error in errors:
f.write(str(error) + "\n")
print(f"Errors logged to errors.txt")
# Print all results to a log file
with open("results.txt", "w", encoding="utf-8") as f:
for result in results:
f.write(result + "\n")
print("All articles pulled successfully.")
except Exception as e:
print(f"Major error in main loop: {e}")
# Continue to next iteration even if there's a major error
# Sleep for a while before the next iteration
print("Sleeping for 15 minutes before the next iteration...")
time.sleep(15 * 60) # Sleep for 15 minutes
if __name__ == "__main__":
main()