New Readmes, cleaned reqs.txt, fixed multithreading issue, and commenting out aiserver and embedder until more testing done
This commit is contained in:
parent
e207823427
commit
f32902e894
103
MCPServer/README.md
Normal file
103
MCPServer/README.md
Normal file
@ -0,0 +1,103 @@
|
|||||||
|
# MCPServer
|
||||||
|
|
||||||
|
A Flask-based server that provides an API for accessing and analyzing financial data, with integration for stock analysis and news processing.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The MCPServer (Model Calling Protocol Server) is designed to provide an API interface for financial data processing including stock market analysis, news aggregation, and financial metrics calculation. It serves as the backend service for processing financial information and making it available through HTTP endpoints.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
MCPServer/
|
||||||
|
├── app.py # Main Flask application
|
||||||
|
├── config.py # Configuration settings
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── Dockerfile # Docker configuration
|
||||||
|
├── README.md # This file
|
||||||
|
└── api/ # API endpoints and handlers
|
||||||
|
├── __init__.py
|
||||||
|
├── stock_analysis.py # Stock analysis functions
|
||||||
|
└── news_processing.py # News processing functions
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Stock Analysis**: Financial metrics calculation and market data processing
|
||||||
|
- **News Integration**: APIs for retrieving and processing financial news
|
||||||
|
- **Data Aggregation**: Consolidation of multiple data sources into unified responses
|
||||||
|
- **RESTful API**: Clean HTTP interface for external services to consume data
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
### Stock Analysis
|
||||||
|
- GET `/api/stock/metrics` - Get financial metrics for a stock
|
||||||
|
- GET `/api/stock/history` - Get historical price data
|
||||||
|
- POST `/api/stock/analyze` - Perform comprehensive stock analysis
|
||||||
|
|
||||||
|
### News Processing
|
||||||
|
- GET `/api/news` - Retrieve news articles related to stocks
|
||||||
|
- GET `/api/news/outlets` - List available news sources
|
||||||
|
- POST `/api/news/process` - Process and categorize news content
|
||||||
|
|
||||||
|
### Data Management
|
||||||
|
- GET `/api/data/refresh` - Refresh data from sources
|
||||||
|
- GET `/api/status` - Server health check
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
The server supports configuration through environment variables:
|
||||||
|
|
||||||
|
- `FLASK_ENV` - Set to 'development' or 'production' (default: 'development')
|
||||||
|
- `DATABASE_URL` - URL for database connection (e.g., PostgreSQL)
|
||||||
|
- `API_KEY` - API key for external services
|
||||||
|
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Install dependencies:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Set up environment variables (optional but recommended):
|
||||||
|
```bash
|
||||||
|
export FLASK_ENV=production
|
||||||
|
export DATABASE_URL="postgresql://user:password@localhost/dbname"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the server:
|
||||||
|
```bash
|
||||||
|
python app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Get stock metrics:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5000/api/stock/metrics?symbol=AAPL"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get historical data:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5000/api/stock/history?symbol=AAPL&days=30"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Process news for a stock:
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:5000/api/news/process" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"symbol":"AAPL","articles":["/path/to/article1.txt","/path/to/article2.txt"]}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6+
|
||||||
|
- Flask 2.3.3
|
||||||
|
- Additional dependencies listed in `requirements.txt`
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License.
|
||||||
@ -1,119 +1,9 @@
|
|||||||
annotated-types==0.7.0
|
Flask
|
||||||
anyio==4.9.0
|
requests
|
||||||
attrs==25.3.0
|
numpy
|
||||||
backoff==2.2.1
|
pandas
|
||||||
bcrypt==4.3.0
|
scikit-learn
|
||||||
blinker==1.9.0
|
sentence-transformers
|
||||||
build==1.2.2.post1
|
chromadb
|
||||||
cachetools==5.5.2
|
transformers
|
||||||
certifi==2025.6.15
|
torch
|
||||||
charset-normalizer==3.4.2
|
|
||||||
chromadb==1.0.13
|
|
||||||
click==8.2.1
|
|
||||||
coloredlogs==15.0.1
|
|
||||||
distro==1.9.0
|
|
||||||
durationpy==0.10
|
|
||||||
einops==0.8.1
|
|
||||||
filelock==3.18.0
|
|
||||||
Flask==3.1.1
|
|
||||||
flatbuffers==25.2.10
|
|
||||||
fsspec==2025.5.1
|
|
||||||
google-auth==2.40.3
|
|
||||||
googleapis-common-protos==1.70.0
|
|
||||||
grpcio==1.73.1
|
|
||||||
h11==0.16.0
|
|
||||||
hf-xet==1.1.5
|
|
||||||
httpcore==1.0.9
|
|
||||||
httptools==0.6.4
|
|
||||||
httpx==0.28.1
|
|
||||||
huggingface-hub==0.33.2
|
|
||||||
humanfriendly==10.0
|
|
||||||
idna==3.10
|
|
||||||
importlib_metadata==8.7.0
|
|
||||||
importlib_resources==6.5.2
|
|
||||||
itsdangerous==2.2.0
|
|
||||||
Jinja2==3.1.6
|
|
||||||
joblib==1.5.1
|
|
||||||
jsonschema==4.24.0
|
|
||||||
jsonschema-specifications==2025.4.1
|
|
||||||
kubernetes==33.1.0
|
|
||||||
markdown-it-py==3.0.0
|
|
||||||
MarkupSafe==3.0.2
|
|
||||||
mdurl==0.1.2
|
|
||||||
mmh3==5.1.0
|
|
||||||
mpmath==1.3.0
|
|
||||||
networkx==3.5
|
|
||||||
numpy==2.3.1
|
|
||||||
nvidia-cublas-cu12==12.6.4.1
|
|
||||||
nvidia-cuda-cupti-cu12==12.6.80
|
|
||||||
nvidia-cuda-nvrtc-cu12==12.6.77
|
|
||||||
nvidia-cuda-runtime-cu12==12.6.77
|
|
||||||
nvidia-cudnn-cu12==9.5.1.17
|
|
||||||
nvidia-cufft-cu12==11.3.0.4
|
|
||||||
nvidia-cufile-cu12==1.11.1.6
|
|
||||||
nvidia-curand-cu12==10.3.7.77
|
|
||||||
nvidia-cusolver-cu12==11.7.1.2
|
|
||||||
nvidia-cusparse-cu12==12.5.4.2
|
|
||||||
nvidia-cusparselt-cu12==0.6.3
|
|
||||||
nvidia-nccl-cu12==2.26.2
|
|
||||||
nvidia-nvjitlink-cu12==12.6.85
|
|
||||||
nvidia-nvtx-cu12==12.6.77
|
|
||||||
oauthlib==3.3.1
|
|
||||||
onnxruntime==1.22.0
|
|
||||||
opentelemetry-api==1.34.1
|
|
||||||
opentelemetry-exporter-otlp-proto-common==1.34.1
|
|
||||||
opentelemetry-exporter-otlp-proto-grpc==1.34.1
|
|
||||||
opentelemetry-proto==1.34.1
|
|
||||||
opentelemetry-sdk==1.34.1
|
|
||||||
opentelemetry-semantic-conventions==0.55b1
|
|
||||||
orjson==3.10.18
|
|
||||||
overrides==7.7.0
|
|
||||||
packaging==25.0
|
|
||||||
pillow==11.3.0
|
|
||||||
posthog==6.0.1
|
|
||||||
protobuf==5.29.5
|
|
||||||
pyasn1==0.6.1
|
|
||||||
pyasn1_modules==0.4.2
|
|
||||||
pybase64==1.4.1
|
|
||||||
pydantic==2.11.7
|
|
||||||
pydantic_core==2.33.2
|
|
||||||
Pygments==2.19.2
|
|
||||||
PyPika==0.48.9
|
|
||||||
pyproject_hooks==1.2.0
|
|
||||||
python-dateutil==2.9.0.post0
|
|
||||||
python-dotenv==1.1.1
|
|
||||||
PyYAML==6.0.2
|
|
||||||
referencing==0.36.2
|
|
||||||
regex==2024.11.6
|
|
||||||
requests==2.32.4
|
|
||||||
requests-oauthlib==2.0.0
|
|
||||||
rich==14.0.0
|
|
||||||
rpds-py==0.26.0
|
|
||||||
rsa==4.9.1
|
|
||||||
safetensors==0.5.3
|
|
||||||
scikit-learn==1.7.0
|
|
||||||
scipy==1.16.0
|
|
||||||
sentence-transformers==5.0.0
|
|
||||||
setuptools==80.9.0
|
|
||||||
shellingham==1.5.4
|
|
||||||
six==1.17.0
|
|
||||||
sniffio==1.3.1
|
|
||||||
sympy==1.14.0
|
|
||||||
tenacity==9.1.2
|
|
||||||
threadpoolctl==3.6.0
|
|
||||||
tokenizers==0.21.2
|
|
||||||
torch==2.7.1
|
|
||||||
tqdm==4.67.1
|
|
||||||
transformers==4.53.0
|
|
||||||
triton==3.3.1
|
|
||||||
typer==0.16.0
|
|
||||||
typing-inspection==0.4.1
|
|
||||||
typing_extensions==4.14.0
|
|
||||||
urllib3==2.5.0
|
|
||||||
uvicorn==0.35.0
|
|
||||||
uvloop==0.21.0
|
|
||||||
watchfiles==1.1.0
|
|
||||||
websocket-client==1.8.0
|
|
||||||
websockets==15.0.1
|
|
||||||
Werkzeug==3.1.3
|
|
||||||
zipp==3.23.0
|
|
||||||
|
|||||||
0
MCPServer/requirements_clean.txt
Normal file
0
MCPServer/requirements_clean.txt
Normal file
151
README.md
Normal file
151
README.md
Normal file
@ -0,0 +1,151 @@
|
|||||||
|
# StockDocs
|
||||||
|
|
||||||
|
A comprehensive financial news analysis platform that combines web scraping, AI processing, and data embedding to provide actionable insights from financial news sources.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
StockDocs is a sophisticated system designed to collect, process, and analyze financial news from multiple sources. The platform consists of several interconnected components that work together to transform raw news content into valuable market intelligence.
|
||||||
|
|
||||||
|
## Project Components
|
||||||
|
|
||||||
|
### 1. Scraper
|
||||||
|
- Web scraping system for collecting financial news via RSS feeds
|
||||||
|
- Organizes articles by news source in a structured directory hierarchy
|
||||||
|
- Supports 60+ financial news outlets including Reuters, Bloomberg, Forbes, and more
|
||||||
|
|
||||||
|
### 2. Article Server
|
||||||
|
- Flask-based HTTP server providing access to collected articles
|
||||||
|
- Query articles by time range and news outlet filters
|
||||||
|
- Retrieve full article content by file path
|
||||||
|
- Exposes RESTful API for external applications
|
||||||
|
|
||||||
|
### 3. AI Processor
|
||||||
|
- Natural language processing engine for analyzing news content
|
||||||
|
- Performs sentiment analysis, topic classification, and entity extraction
|
||||||
|
- Generates actionable insights from financial articles
|
||||||
|
- Supports batch processing of large volumes of content
|
||||||
|
|
||||||
|
### 4. Embedding Service
|
||||||
|
- Converts text content into numerical vector representations
|
||||||
|
- Enables semantic similarity comparisons between articles
|
||||||
|
- Supports various transformer-based models for high-quality embeddings
|
||||||
|
- Provides caching mechanism to optimize performance
|
||||||
|
|
||||||
|
### 5. MCPServer
|
||||||
|
- Market Capitalization Processor Server
|
||||||
|
- Provides financial data processing and API interface
|
||||||
|
- Integrates with stock analysis and news processing functions
|
||||||
|
- Serves as backend service for external access to financial data
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
+--------------+ +--------------+ +-------------------+
|
||||||
|
| Scraper | | Article | | AI |
|
||||||
|
| (RSS Feeds) |--> | Server |--> | Processor |
|
||||||
|
| | | | | |
|
||||||
|
+--------------+ +--------------+ +-------------------+
|
||||||
|
| |
|
||||||
|
v v
|
||||||
|
+--------------+ +-------------------+
|
||||||
|
| Embedding | | MCPServer |
|
||||||
|
| Service | | (Financial Data) |
|
||||||
|
| | | |
|
||||||
|
+--------------+ +-------------------+
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Multi-source News Collection**: Aggregates content from major financial news outlets
|
||||||
|
- **Real-time Processing**: Automated scraping and analysis pipeline
|
||||||
|
- **Advanced Analytics**: NLP-powered sentiment and topic analysis
|
||||||
|
- **Semantic Search**: Vector-based similarity comparisons
|
||||||
|
- **RESTful APIs**: Clean interfaces for integration with external systems
|
||||||
|
- **Containerized Deployment**: Docker support for easy deployment
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- Python 3.6+
|
||||||
|
- Docker (for containerized deployment)
|
||||||
|
- Internet connection for RSS feed access
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
1. Clone the repository:
|
||||||
|
```bash
|
||||||
|
git clone <repository-url>
|
||||||
|
cd StockDocs
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Set up each component:
|
||||||
|
```bash
|
||||||
|
# For each component, follow specific installation instructions
|
||||||
|
cd scraper && pip install -r requirements.txt
|
||||||
|
cd articleServer && pip install -r requirements.txt
|
||||||
|
cd ai_processor && pip install -r requirements.txt
|
||||||
|
cd embedding && pip install -r requirements.txt
|
||||||
|
cd MCPServer && pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Configure environment variables as needed for each component
|
||||||
|
|
||||||
|
4. Run individual services:
|
||||||
|
```bash
|
||||||
|
python scraper/scraper.py # Start scraping
|
||||||
|
python articleServer/run_server.py # Start article server
|
||||||
|
python ai_processor/app.py # Start AI processor
|
||||||
|
python embedding/app.py # Start embedding service
|
||||||
|
python MCPServer/app.py # Start MCP server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Data Collection
|
||||||
|
The scraper component automatically collects news from configured RSS feeds and stores articles in structured directories.
|
||||||
|
|
||||||
|
### API Access
|
||||||
|
Use the article server's RESTful APIs to access collected content:
|
||||||
|
```bash
|
||||||
|
# Get recent articles
|
||||||
|
curl "http://localhost:5008/articles?time_range=hour"
|
||||||
|
|
||||||
|
# Get article content
|
||||||
|
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Analysis
|
||||||
|
The AI processor and embedding service provide advanced analysis capabilities through their respective APIs.
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
Each component can be run independently or containerized using the provided Dockerfiles:
|
||||||
|
```bash
|
||||||
|
# Build and run each component in Docker
|
||||||
|
docker build -t stockdocs-scraper ./scraper
|
||||||
|
docker run -p 5000:5000 stockdocs-scraper
|
||||||
|
|
||||||
|
docker build -t stockdocs-article-server ./articleServer
|
||||||
|
docker run -p 5008:5008 stockdocs-article-server
|
||||||
|
|
||||||
|
# Continue for other components...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6+
|
||||||
|
- Flask 2.3.3
|
||||||
|
- Various NLP and ML libraries
|
||||||
|
- Docker (for containerized deployment)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
Contributions are welcome! Please read our contribution guidelines before submitting pull requests.
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
For support, please open an issue on the GitHub repository.
|
||||||
118
ai_processor/README.md
Normal file
118
ai_processor/README.md
Normal file
@ -0,0 +1,118 @@
|
|||||||
|
# AI Processor
|
||||||
|
|
||||||
|
An AI-powered analytics engine designed to process financial news articles and extract meaningful insights, sentiment analysis, and market indicators from collected content.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The AI Processor is the intelligent component of the system that analyzes the financial news articles collected by the scraper. It uses natural language processing techniques and machine learning models to extract key insights, determine sentiment, identify market trends, and generate actionable analytics for traders and investors.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
ai_processor/
|
||||||
|
├── app.py # Main AI processing application
|
||||||
|
├── config.py # Configuration settings
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── Dockerfile # Docker configuration
|
||||||
|
├── README.md # This file
|
||||||
|
├── models/ # Machine learning models and NLP components
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── sentiment_analyzer.py # Sentiment analysis module
|
||||||
|
│ ├── topic_classifier.py # News categorization module
|
||||||
|
│ └── entity_extractor.py # Named entity recognition
|
||||||
|
├── processors/ # Article processing pipelines
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── text_processor.py # Text cleaning and preprocessing
|
||||||
|
│ └── analysis_pipeline.py # Full analysis pipeline
|
||||||
|
└── data/ # Processed data storage
|
||||||
|
├── insights/
|
||||||
|
└── reports/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Sentiment Analysis**: Determine positive, negative, or neutral sentiment of news articles
|
||||||
|
- **Topic Classification**: Categorize articles by financial topics (economics, politics, technology, etc.)
|
||||||
|
- **Entity Extraction**: Identify key entities mentioned in articles (stocks, companies, people, organizations)
|
||||||
|
- **Market Indicator Detection**: Extract quantitative indicators that may affect stock prices
|
||||||
|
- **Insight Generation**: Automated generation of actionable intelligence from news content
|
||||||
|
- **Batch Processing**: Process large volumes of articles efficiently
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
### Article Analysis
|
||||||
|
- POST `/api/analyze/article` - Analyze a single article for insights
|
||||||
|
- POST `/api/analyze/batch` - Process multiple articles in batch mode
|
||||||
|
- GET `/api/analyze/status/{task_id}` - Check processing status
|
||||||
|
|
||||||
|
### Data Access
|
||||||
|
- GET `/api/insights/latest` - Get latest analysis insights
|
||||||
|
- GET `/api/insights/articles/{article_path}` - Get insights for specific article
|
||||||
|
- GET `/api/reports/generate` - Generate comprehensive market analysis report
|
||||||
|
|
||||||
|
### Model Management
|
||||||
|
- GET `/api/models` - List available AI models
|
||||||
|
- POST `/api/models/update` - Update or retrain models with new data
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
The AI Processor supports configuration through environment variables:
|
||||||
|
|
||||||
|
- `MODEL_PATH` - Path to pre-trained AI models (default: `models/`)
|
||||||
|
- `ARTICLE_DIR` - Directory containing articles to process (default: `../scraper/articles`)
|
||||||
|
- `ENABLE_CACHING` - Enable/disable result caching (default: True)
|
||||||
|
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||||
|
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Install dependencies:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Set up environment variables (optional but recommended):
|
||||||
|
```bash
|
||||||
|
export MODEL_PATH="/path/to/models"
|
||||||
|
export ARTICLE_DIR="/path/to/articles"
|
||||||
|
export ENABLE_CACHING=true
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the AI processor:
|
||||||
|
```bash
|
||||||
|
python app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Analyze a single article:
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:5001/api/analyze/article" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"path":"/path/to/article.txt","source":"Reuters – Business News"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch process articles:
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:5001/api/analyze/batch" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"article_paths":["/path/to/article1.txt","/path/to/article2.txt"],"include_sentiment":true}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get latest insights:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5001/api/insights/latest?limit=10"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6+
|
||||||
|
- NLP libraries (spaCy, NLTK, transformers)
|
||||||
|
- Machine learning frameworks (scikit-learn, tensorflow/PyTorch)
|
||||||
|
- Additional dependencies listed in `requirements.txt`
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License.
|
||||||
@ -1,5 +0,0 @@
|
|||||||
certifi==2025.7.14
|
|
||||||
charset-normalizer==3.4.2
|
|
||||||
idna==3.10
|
|
||||||
requests==2.32.4
|
|
||||||
urllib3==2.5.0
|
|
||||||
0
ai_processor/requirements_clean.txt
Normal file
0
ai_processor/requirements_clean.txt
Normal file
192
articleServer/README.md
Normal file
192
articleServer/README.md
Normal file
@ -0,0 +1,192 @@
|
|||||||
|
# Article Server
|
||||||
|
|
||||||
|
A Flask-based HTTP server that provides access to news articles stored in a directory structure, with time-based filtering and outlet-specific querying capabilities.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This server allows you to query news articles from various sources based on:
|
||||||
|
- Time range (last hour, day, week, month)
|
||||||
|
- Specific news outlets
|
||||||
|
- Direct content retrieval by file path
|
||||||
|
|
||||||
|
Articles are organized in a nested directory structure where each news outlet has its own subdirectory containing the respective articles.
|
||||||
|
|
||||||
|
## Directory Structure
|
||||||
|
|
||||||
|
The server expects the following directory structure:
|
||||||
|
```
|
||||||
|
scraper/
|
||||||
|
└── articles/
|
||||||
|
├── Reuters – Business News/
|
||||||
|
│ ├── article1.txt
|
||||||
|
│ ├── article2.txt
|
||||||
|
│ └── ...
|
||||||
|
├── Associated Press – Business/
|
||||||
|
│ ├── article1.txt
|
||||||
|
│ └── ...
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
### 1. Get Articles (`/articles`)
|
||||||
|
Retrieve articles within a specified time range.
|
||||||
|
|
||||||
|
**Method:** `GET`
|
||||||
|
**Parameters:**
|
||||||
|
- `time_range` (optional): hour, day, week, month (default: hour)
|
||||||
|
- `outlets` (optional): comma-separated list of news outlet names
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
# Get articles from last day for specific outlets
|
||||||
|
curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"
|
||||||
|
|
||||||
|
# Get all articles from the last hour
|
||||||
|
curl "http://localhost:5008/articles"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"articles": [
|
||||||
|
{
|
||||||
|
"path": "/path/to/article.txt",
|
||||||
|
"name": "article.txt",
|
||||||
|
"outlet": "Reuters – Business News",
|
||||||
|
"created_at": "2023-10-17T14:30:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"count": 5,
|
||||||
|
"time_range": "hour",
|
||||||
|
"outlets": ["Reuters – Business News"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Get Article Content (`/article/content`)
|
||||||
|
Retrieve the full content of a specific article by file path.
|
||||||
|
|
||||||
|
**Method:** `GET`
|
||||||
|
**Parameters:**
|
||||||
|
- `path` (required): Absolute path to the article file
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
# Get full content of an article
|
||||||
|
curl "http://localhost:5008/article/content?path=/full/path/to/article.txt"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"path": "/full/path/to/article.txt",
|
||||||
|
"name": "article.txt",
|
||||||
|
"outlet": "Reuters – Business News",
|
||||||
|
"content": "Full article content here..."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Get Available Outlets (`/outlets`)
|
||||||
|
List all available news outlets.
|
||||||
|
|
||||||
|
**Method:** `GET`
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/outlets"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"news_outlets": [
|
||||||
|
"Reuters – Business News",
|
||||||
|
"Associated Press – Business",
|
||||||
|
"Financial Times",
|
||||||
|
...
|
||||||
|
],
|
||||||
|
"count": 60
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Health Check (`/health`)
|
||||||
|
Simple health check endpoint.
|
||||||
|
|
||||||
|
**Method:** `GET`
|
||||||
|
**Example:**
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/health"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "healthy"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
- `ARTICLE_DIR` (optional): Path to the article directory. Defaults to `scraper/articles` if not set.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Install dependencies:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Set the article directory path (optional):
|
||||||
|
```bash
|
||||||
|
export ARTICLE_DIR="/path/to/your/articles"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the server:
|
||||||
|
```bash
|
||||||
|
python run_server.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Or with Docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t article-server .
|
||||||
|
docker run -p 5008:5008 article-server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Get recent articles from all outlets:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/articles?time_range=hour"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get articles from the last day for specific outlets:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get all available news outlets:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/outlets"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Retrieve full content of a specific article:
|
||||||
|
```bash
|
||||||
|
curl "http://localhost:5008/article/content?path=/absolute/path/to/your/article.txt"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
- The server validates that all requested file paths are within the configured article directory to prevent directory traversal attacks
|
||||||
|
- All file paths must be absolute and within the allowed directory structure
|
||||||
|
- Path parameters are URL decoded for proper handling of special characters
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6+
|
||||||
|
- Flask 2.3.3
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License.
|
||||||
@ -21,30 +21,30 @@ services:
|
|||||||
- "5005:5005"
|
- "5005:5005"
|
||||||
environment:
|
environment:
|
||||||
- FLASK_ENV=development
|
- FLASK_ENV=development
|
||||||
ai_processor:
|
# ai_processor:
|
||||||
build: ./ai_processor
|
# build: ./ai_processor
|
||||||
platform: linux/amd64
|
# platform: linux/amd64
|
||||||
container_name: stockdocs-ai-processor
|
# container_name: stockdocs-ai-processor
|
||||||
restart: unless-stopped
|
# restart: unless-stopped
|
||||||
networks:
|
# networks:
|
||||||
- ainetwork
|
# - ainetwork
|
||||||
volumes:
|
# volumes:
|
||||||
- ./scraper/articles:/app/articles
|
# - ./scraper/articles:/app/articles
|
||||||
- ./ai_processor/output:/app/output
|
# - ./ai_processor/output:/app/output
|
||||||
environment:
|
# environment:
|
||||||
- AI_SERVICE_URL=http://192.168.8.124:11434 # Local AI service IP
|
# - AI_SERVICE_URL=http://192.168.8.124:11434 # Local AI service IP
|
||||||
embedder:
|
# embedder:
|
||||||
build: ./embedding
|
# build: ./embedding
|
||||||
platform: linux/amd64
|
# platform: linux/amd64
|
||||||
container_name: stockdocs-embedder
|
# container_name: stockdocs-embedder
|
||||||
restart: unless-stopped
|
# restart: unless-stopped
|
||||||
networks:
|
# networks:
|
||||||
- ainetwork
|
# - ainetwork
|
||||||
volumes:
|
# volumes:
|
||||||
- ./ai_processor/output:/app/output
|
# - ./ai_processor/output:/app/output
|
||||||
environment:
|
# environment:
|
||||||
- CHROMADB_HOST=chromadb
|
# - CHROMADB_HOST=chromadb
|
||||||
- CHROMADB_PORT=8000
|
# - CHROMADB_PORT=8000
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
ainetwork:
|
ainetwork:
|
||||||
|
|||||||
113
embedding/README.md
Normal file
113
embedding/README.md
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
# Embedding
|
||||||
|
|
||||||
|
An embedding service that converts text content into numerical vectors for machine learning and data analysis purposes. This component transforms financial news articles into vector representations that can be used for similarity comparisons, clustering, and other AI tasks.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The Embedding project provides a service for converting textual financial news content into dense vector representations (embeddings) using state-of-the-art natural language processing models. These embeddings capture semantic meaning and relationships between different pieces of financial content, enabling advanced analytics and machine learning applications.
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
embedding/
|
||||||
|
├── app.py # Main embedding application
|
||||||
|
├── config.py # Configuration settings
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
├── Dockerfile # Docker configuration
|
||||||
|
├── README.md # This file
|
||||||
|
├── models/ # Pre-trained embedding models
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── sentence_transformer.py # Sentence transformer implementation
|
||||||
|
│ └── model_loader.py # Model loading utilities
|
||||||
|
├── processors/ # Text processing pipeline
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── text_cleaner.py # Text cleaning and preprocessing
|
||||||
|
│ └── embedding_generator.py # Embedding generation
|
||||||
|
└── data/ # Processed embeddings storage
|
||||||
|
├── cache/
|
||||||
|
└── outputs/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Multiple Model Support**: Integrates with various embedding models including BERT, Sentence-BERT, and other transformer-based models
|
||||||
|
- **Batch Processing**: Efficient processing of large volumes of articles
|
||||||
|
- **Caching Mechanism**: Caches generated embeddings to avoid reprocessing
|
||||||
|
- **Real-time Generation**: Generates embeddings on-demand for new content
|
||||||
|
- **Vector Similarity**: Computes similarity between different pieces of content
|
||||||
|
- **Storage Management**: Organizes and stores embeddings efficiently
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
### Embedding Generation
|
||||||
|
- POST `/api/embeddings/generate` - Generate embeddings for text content
|
||||||
|
- POST `/api/embeddings/batch` - Generate embeddings for multiple texts in batch
|
||||||
|
- GET `/api/embeddings/similarity` - Calculate similarity between two pieces of text/content
|
||||||
|
|
||||||
|
### Data Management
|
||||||
|
- GET `/api/embeddings/status` - Check service status
|
||||||
|
- GET `/api/embeddings/models` - List available embedding models
|
||||||
|
- DELETE `/api/embeddings/cache/clear` - Clear the embedding cache
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
The embedding service supports configuration through environment variables:
|
||||||
|
|
||||||
|
- `EMBEDDING_MODEL` - Name of the pre-trained model to use (default: "all-MiniLM-L6-v2")
|
||||||
|
- `CACHE_DIR` - Directory for caching generated embeddings (default: "data/cache/")
|
||||||
|
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
|
||||||
|
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||||
|
- `EMBEDDING_DIMENSION` - Dimension size for generated vectors (default: 384)
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Install dependencies:
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Set up environment variables (optional but recommended):
|
||||||
|
```bash
|
||||||
|
export EMBEDDING_MODEL="all-MiniLM-L6-v2"
|
||||||
|
export CACHE_DIR="./data/cache/"
|
||||||
|
export MAX_WORKERS=4
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Run the embedding service:
|
||||||
|
```bash
|
||||||
|
python app.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Generate embeddings for a single text:
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:5002/api/embeddings/generate" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"text":"The stock market showed strong performance today.","source":"Reuters"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch processing of multiple texts:
|
||||||
|
```bash
|
||||||
|
curl -X POST "http://localhost:5002/api/embeddings/batch" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"texts":["Article 1 content","Article 2 content"],"source":"Financial News"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Calculate similarity between two pieces of text:
|
||||||
|
```bash
|
||||||
|
curl -X GET "http://localhost:5002/api/embeddings/similarity?text1=stock%20market&text2=financial%20market"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.6+
|
||||||
|
- Transformer models (transformers, sentence-transformers)
|
||||||
|
- Vector processing libraries (numpy, scikit-learn)
|
||||||
|
- Additional dependencies listed in `requirements.txt`
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License.
|
||||||
@ -1,4 +0,0 @@
|
|||||||
chromadb==1.0.13
|
|
||||||
einops==0.8.1
|
|
||||||
requests==2.31.0
|
|
||||||
sentence-transformers==5.0.0
|
|
||||||
@ -1,5 +1,5 @@
|
|||||||
home = /usr/bin
|
home = /opt/homebrew/opt/python@3.13/bin
|
||||||
include-system-site-packages = false
|
include-system-site-packages = false
|
||||||
version = 3.12.3
|
version = 3.13.7
|
||||||
executable = /usr/bin/python3.12
|
executable = /opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/bin/python3.13
|
||||||
command = /usr/bin/python3 -m venv /home/user/scraper
|
command = /opt/homebrew/opt/python@3.13/bin/python3.13 -m venv /Users/user/Projects/StockDocs/scraper
|
||||||
|
|||||||
@ -1,44 +1,43 @@
|
|||||||
attrs==25.3.0
|
attrs
|
||||||
beautifulsoup4==4.13.4
|
beautifulsoup4
|
||||||
certifi==2025.6.15
|
certifi
|
||||||
charset-normalizer==3.4.2
|
charset-normalizer
|
||||||
click==8.2.1
|
click
|
||||||
dnspython==2.7.0
|
dnspython
|
||||||
feedparser==6.0.11
|
feedparser
|
||||||
filelock==3.18.0
|
filelock
|
||||||
gnews==0.4.1
|
gnews
|
||||||
greenlet==3.2.3
|
greenlet
|
||||||
h11==0.16.0
|
h11
|
||||||
idna==3.10
|
idna
|
||||||
joblib==1.5.1
|
joblib
|
||||||
lxml==5.4.0
|
lxml
|
||||||
lxml-html-clean==0.4.2
|
lxml-html-clean
|
||||||
newspaper4k==0.9.3.1
|
newspaper4k
|
||||||
nltk==3.9.1
|
nltk
|
||||||
numpy==2.3.0
|
numpy
|
||||||
outcome==1.3.0.post0
|
outcome
|
||||||
pandas==2.3.0
|
pandas
|
||||||
pillow==11.2.1
|
pillow
|
||||||
playwright==1.52.0
|
playwright
|
||||||
pyee==13.0.0
|
pyee
|
||||||
pysocks==1.7.1
|
pysocks
|
||||||
python-dateutil==2.9.0.post0
|
python-dateutil
|
||||||
pytz==2025.2
|
pytz
|
||||||
pyyaml==6.0.2
|
pyyaml
|
||||||
regex==2024.11.6
|
regex
|
||||||
requests==2.32.4
|
requests
|
||||||
requests-file==2.1.0
|
requests-file
|
||||||
selenium==4.33.0
|
selenium
|
||||||
sgmllib3k==1.0.0
|
sgmllib3k
|
||||||
six==1.17.0
|
six
|
||||||
sniffio==1.3.1
|
sniffio
|
||||||
sortedcontainers==2.4.0
|
sortedcontainers
|
||||||
soupsieve==2.7
|
soupsieve
|
||||||
tldextract==5.3.0
|
tldextract
|
||||||
tqdm==4.67.1
|
tqdm
|
||||||
trio==0.30.0
|
trio
|
||||||
trio-websocket==0.12.2
|
trio-websocket
|
||||||
tzdata==2025.2
|
tzdata
|
||||||
websocket-client==1.8.0
|
websocket-client
|
||||||
wsproto==1.2.0
|
wsproto
|
||||||
|
|
||||||
|
|||||||
@ -1,13 +1,8 @@
|
|||||||
{
|
{
|
||||||
"rss_feeds": {
|
"rss_feeds": {
|
||||||
"Reuters – Business News": {
|
|
||||||
"source_website": "reuters.com",
|
|
||||||
"rss_url": "https://news.google.com/rss/search?q=site:reuters.com+business&hl=en-US&gl=US&ceid=US:en"
|
|
||||||
},
|
|
||||||
"Associated Press – Business": {
|
"Associated Press – Business": {
|
||||||
"source_website": "apnews.com",
|
"source_website": "apnews.com",
|
||||||
"rss_url": "https://news.google.com/rss/search?q=site:apnews.com&hl=en-US&gl=US&ceid=US:en"
|
"rss_url": "https://news.google.com/rss/search?q=site:apnews.com&hl=en-US&gl=US&ceid=US:en"
|
||||||
}
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
|
|||||||
@ -5,20 +5,23 @@ import time
|
|||||||
import os
|
import os
|
||||||
from selenium import webdriver
|
from selenium import webdriver
|
||||||
from selenium.webdriver.firefox.options import Options as FirefoxOptions
|
from selenium.webdriver.firefox.options import Options as FirefoxOptions
|
||||||
from newspaper.google_news import GoogleNewsSource
|
from selenium.webdriver.common.by import By
|
||||||
from concurrent.futures import ProcessPoolExecutor
|
from selenium.webdriver.support.ui import WebDriverWait
|
||||||
|
from selenium.webdriver.support import expected_conditions as EC
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
import nltk
|
import nltk
|
||||||
from nltk.downloader import Downloader
|
from nltk.downloader import Downloader
|
||||||
|
|
||||||
FEED_FILE = os.getenv("FEED_FILE")
|
FEED_FILE = os.getenv("FEED_FILE", "./rss_short_feed.json")
|
||||||
|
|
||||||
# Ensure necessary NLTK resources are downloaded / Needed for selenium + newspaper4k article parsing
|
# Ensure necessary NLTK resources are downloaded
|
||||||
d = Downloader()
|
d = Downloader()
|
||||||
if not d.is_installed('punkt_tab'):
|
if not d.is_installed("punkt_tab"):
|
||||||
nltk.download('punkt_tab')
|
nltk.download("punkt_tab")
|
||||||
|
|
||||||
articles = []
|
articles = []
|
||||||
|
|
||||||
|
|
||||||
def load_rss_feed_sources(feed_file=FEED_FILE):
|
def load_rss_feed_sources(feed_file=FEED_FILE):
|
||||||
"""
|
"""
|
||||||
Loads the RSS feed sources from a JSON file.
|
Loads the RSS feed sources from a JSON file.
|
||||||
@ -35,13 +38,14 @@ def load_rss_feed_sources(feed_file=FEED_FILE):
|
|||||||
print("Error decoding " + FEED_FILE + " , returning empty list.")
|
print("Error decoding " + FEED_FILE + " , returning empty list.")
|
||||||
return []
|
return []
|
||||||
|
|
||||||
|
|
||||||
def mine_all_articles(rss_feed_sources, limit=None):
|
def mine_all_articles(rss_feed_sources, limit=None):
|
||||||
"""
|
"""
|
||||||
Mines all articles from the given RSS feed sources.
|
Mines all articles from the given RSS feed sources.
|
||||||
Returns a list of (site, title, link) tuples.
|
Returns a list of (site, title, link) tuples.
|
||||||
"""
|
"""
|
||||||
all_links = []
|
all_links = []
|
||||||
sources = rss_feed_sources['rss_feeds']
|
sources = rss_feed_sources["rss_feeds"]
|
||||||
|
|
||||||
for site, data in sources.items():
|
for site, data in sources.items():
|
||||||
print(f"Parsing RSS feed: {data['rss_url']}")
|
print(f"Parsing RSS feed: {data['rss_url']}")
|
||||||
@ -56,6 +60,7 @@ def mine_all_articles(rss_feed_sources, limit=None):
|
|||||||
print(f"Error parsing RSS feed: {site} Error: {str(e)}")
|
print(f"Error parsing RSS feed: {site} Error: {str(e)}")
|
||||||
return all_links
|
return all_links
|
||||||
|
|
||||||
|
|
||||||
def generate_filename_from_url(url):
|
def generate_filename_from_url(url):
|
||||||
"""
|
"""
|
||||||
Generates a filename from the given URL by replacing slashes with underscores.
|
Generates a filename from the given URL by replacing slashes with underscores.
|
||||||
@ -63,12 +68,15 @@ def generate_filename_from_url(url):
|
|||||||
# Use only the last part of the URL or replace slashes
|
# Use only the last part of the URL or replace slashes
|
||||||
return url.replace("https://", "").replace("http://", "").replace("/", "_")
|
return url.replace("https://", "").replace("http://", "").replace("/", "_")
|
||||||
|
|
||||||
|
|
||||||
def generate_safe_filename(name):
|
def generate_safe_filename(name):
|
||||||
# Remove/replace characters not allowed in filenames
|
# Remove/replace characters not allowed in filenames
|
||||||
import re
|
import re
|
||||||
|
|
||||||
safe = re.sub(r'[\\/*?:"<>|]', "_", name)
|
safe = re.sub(r'[\\/*?:"<>|]', "_", name)
|
||||||
return safe
|
return safe
|
||||||
|
|
||||||
|
|
||||||
def save_article_to_file(article, filename, source="Unfiltered"):
|
def save_article_to_file(article, filename, source="Unfiltered"):
|
||||||
"""
|
"""
|
||||||
Saves the given article text to a file with the specified filename.
|
Saves the given article text to a file with the specified filename.
|
||||||
@ -76,7 +84,7 @@ def save_article_to_file(article, filename, source="Unfiltered"):
|
|||||||
# articles dir should already be there
|
# articles dir should already be there
|
||||||
# os.makedirs("articles", exist_ok=True)
|
# os.makedirs("articles", exist_ok=True)
|
||||||
|
|
||||||
outputDir = "articles/"+source
|
outputDir = "articles/" + source
|
||||||
os.makedirs(outputDir, exist_ok=True) if source else None
|
os.makedirs(outputDir, exist_ok=True) if source else None
|
||||||
|
|
||||||
# Sanitize filename: use only the last part of the URL or replace slashes
|
# Sanitize filename: use only the last part of the URL or replace slashes
|
||||||
@ -88,87 +96,140 @@ def save_article_to_file(article, filename, source="Unfiltered"):
|
|||||||
f.write(article)
|
f.write(article)
|
||||||
print(f"Article saved to {file_path}")
|
print(f"Article saved to {file_path}")
|
||||||
|
|
||||||
|
|
||||||
def get_article_with_selenium(url):
|
def get_article_with_selenium(url):
|
||||||
|
"""
|
||||||
|
Gets article text using Selenium Firefox driver with proper error handling
|
||||||
|
and cleanup.
|
||||||
|
"""
|
||||||
|
driver = None
|
||||||
|
try:
|
||||||
|
# Configure Firefox options
|
||||||
options = FirefoxOptions()
|
options = FirefoxOptions()
|
||||||
options.add_argument("--headless")
|
options.add_argument("--headless")
|
||||||
|
options.set_preference("dom.ipc.processCount", 1) # Reduce process count
|
||||||
|
|
||||||
|
# Initialize driver
|
||||||
driver = webdriver.Firefox(options=options)
|
driver = webdriver.Firefox(options=options)
|
||||||
driver.set_page_load_timeout(30) # 30 seconds timeout
|
driver.set_page_load_timeout(30) # 30 seconds timeout
|
||||||
|
|
||||||
|
# Navigate to URL
|
||||||
#driver = webdriver.Remote(
|
|
||||||
#command_executor='http://localhost:4444/wd/hub',
|
|
||||||
#options=options)
|
|
||||||
try:
|
|
||||||
driver.get(url)
|
driver.get(url)
|
||||||
time.sleep(5) # Wait for JS to load
|
|
||||||
|
# Wait for page to load (explicit wait instead of sleep)
|
||||||
|
try:
|
||||||
|
WebDriverWait(driver, 15).until(
|
||||||
|
EC.presence_of_element_located((By.TAG_NAME, "body"))
|
||||||
|
)
|
||||||
|
except:
|
||||||
|
pass # Continue even if wait times out
|
||||||
|
|
||||||
|
time.sleep(2) # Brief additional wait
|
||||||
|
|
||||||
html = driver.page_source
|
html = driver.page_source
|
||||||
|
|
||||||
# Parse with Newspaper4k
|
# Parse with Newspaper4k
|
||||||
article = newspaper.article(url, input_html=html, language='en')
|
article = newspaper.article(url, input_html=html, language="en")
|
||||||
article.nlp()
|
article.nlp()
|
||||||
return article.text
|
return article.text
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Selenium failed for {url}: {str(e)}")
|
||||||
|
return ""
|
||||||
finally:
|
finally:
|
||||||
|
# Always quit the driver
|
||||||
|
if driver:
|
||||||
|
try:
|
||||||
driver.quit()
|
driver.quit()
|
||||||
|
except:
|
||||||
|
pass # Ignore errors in cleanup
|
||||||
|
|
||||||
|
|
||||||
def get_article_with_playwright(url):
|
def get_article_with_playwright(url):
|
||||||
import asyncio
|
"""
|
||||||
|
Gets article text using Playwright with proper error handling.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
from playwright.sync_api import sync_playwright
|
from playwright.sync_api import sync_playwright
|
||||||
from newspaper import Article
|
|
||||||
|
|
||||||
with sync_playwright() as p:
|
with sync_playwright() as p:
|
||||||
browser = p.chromium.launch(headless=True)
|
# Use Chromium instead of Firefox for better compatibility
|
||||||
|
browser = p.chromium.launch(headless=True, timeout=30000)
|
||||||
page = browser.new_page()
|
page = browser.new_page()
|
||||||
page.goto(url)
|
|
||||||
|
|
||||||
# Optional: wait for specific content to load
|
# Set user agent to avoid bot detection
|
||||||
time.sleep(5) # Adjust as needed for the page to load completely
|
page.set_extra_http_headers(
|
||||||
|
{
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
page.goto(url, wait_until="load")
|
||||||
|
|
||||||
|
# Wait for content to load
|
||||||
|
time.sleep(3)
|
||||||
|
|
||||||
html = page.content()
|
html = page.content()
|
||||||
|
|
||||||
browser.close()
|
browser.close()
|
||||||
|
|
||||||
# Parse with Newspaper4k
|
# Parse with Newspaper4k
|
||||||
article = newspaper.article(url, input_html=html, language='en')
|
article = newspaper.article(url, input_html=html, language="en")
|
||||||
article.nlp()
|
article.nlp()
|
||||||
return article.text
|
return article.text
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Playwright failed for {url}: {str(e)}")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
def pull_article(link, source, title=None, save_to_file=True):
|
def pull_article(link, source, title=None, save_to_file=True):
|
||||||
|
"""
|
||||||
|
Pulls an article from a given link with fallback mechanisms.
|
||||||
|
"""
|
||||||
filename = title if title else link
|
filename = title if title else link
|
||||||
safe_filename = generate_filename_from_url(filename)
|
safe_filename = generate_filename_from_url(filename)
|
||||||
|
|
||||||
|
# Check if already cached
|
||||||
if os.path.exists(os.path.join("articles", source, safe_filename)):
|
if os.path.exists(os.path.join("articles", source, safe_filename)):
|
||||||
print(f"Article already cached: {filename}")
|
print(f"Article already cached: {filename}")
|
||||||
with open(os.path.join("articles", source, safe_filename), "r", encoding="utf-8") as f:
|
with open(
|
||||||
|
os.path.join("articles", source, safe_filename), "r", encoding="utf-8"
|
||||||
|
) as f:
|
||||||
return f.read()
|
return f.read()
|
||||||
|
|
||||||
text = ""
|
text = ""
|
||||||
|
|
||||||
time.sleep(5) # Since we spawn lots of processes, we need to sleep at the start
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
# Try newspaper4k first
|
||||||
article = newspaper.article(link)
|
article = newspaper.article(link)
|
||||||
article.download()
|
article.download()
|
||||||
article.parse()
|
article.parse()
|
||||||
text = article.text
|
text = article.text
|
||||||
|
|
||||||
if not text or len(text) < 200:
|
if not text or len(text) < 200:
|
||||||
raise ValueError("\tArticle text too short, falling back to Playwright/Selenium.")
|
raise ValueError(
|
||||||
|
"\tArticle text too short, falling back to Playwright/Selenium."
|
||||||
|
)
|
||||||
print(f"\tSuccessfully pulled article with newspaper4k from {link}")
|
print(f"\tSuccessfully pulled article with newspaper4k from {link}")
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"\tnewspaper4k extraction failed for {link}: {e}, falling back to Playwright.")
|
print(
|
||||||
|
f"\tnewspaper4k extraction failed for {link}: {e}, falling back to Playwright."
|
||||||
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
text = get_article_with_playwright(link)
|
text = get_article_with_playwright(link)
|
||||||
print(f"\t\tSuccessfully pulled article from {link} with Playwright")
|
print(f"\t\tSuccessfully pulled article from {link} with Playwright")
|
||||||
|
|
||||||
if not text or len(text) < 200:
|
if not text or len(text) < 200:
|
||||||
print(f"\t\tPlaywright article too short, falling back to Selenium.")
|
print(f"\t\tPlaywright article too short, falling back to Selenium.")
|
||||||
try:
|
# Fallback to Selenium with better error handling
|
||||||
text = get_article_with_selenium(link)
|
text = get_article_with_selenium(link)
|
||||||
print(f"\t\t\tSuccessfully pulled article from {link} with Selenium")
|
print(f"\t\t\tSuccessfully pulled article from {link} with Selenium")
|
||||||
except Exception as e:
|
|
||||||
print(f"\t\t\tSelenium failed for {link}: {e}")
|
|
||||||
return ""
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"\t\tPlaywright failed for {link}: {e}")
|
print(f"\t\tPlaywright failed for {link}: {e}")
|
||||||
|
|
||||||
|
# Fallback to Selenium
|
||||||
try:
|
try:
|
||||||
text = get_article_with_selenium(link)
|
text = get_article_with_selenium(link)
|
||||||
print(f"\t\tSuccessfully pulled article from {link} with Selenium")
|
print(f"\t\tSuccessfully pulled article from {link} with Selenium")
|
||||||
@ -178,11 +239,55 @@ def pull_article(link, source, title=None, save_to_file=True):
|
|||||||
|
|
||||||
if save_to_file:
|
if save_to_file:
|
||||||
save_article_to_file(text, filename, source)
|
save_article_to_file(text, filename, source)
|
||||||
|
|
||||||
return text
|
return text
|
||||||
|
|
||||||
while True:
|
|
||||||
|
def safe_pull_articles(article_list):
|
||||||
|
"""
|
||||||
|
Safely pull articles with improved error handling and reduced parallelism.
|
||||||
|
"""
|
||||||
|
results = []
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
# Process in smaller batches to reduce resource strain
|
||||||
|
batch_size = 5
|
||||||
|
|
||||||
|
for i in range(0, len(article_list), batch_size):
|
||||||
|
batch = article_list[i : i + batch_size]
|
||||||
|
print(f"Processing batch {i // batch_size + 1} with {len(batch)} articles")
|
||||||
|
|
||||||
|
# Use ThreadPoolExecutor instead of ProcessPoolExecutor to avoid
|
||||||
|
# process termination issues with browser automation
|
||||||
|
with ThreadPoolExecutor(max_workers=3) as executor: # Reduced workers
|
||||||
|
futures = [
|
||||||
|
executor.submit(pull_article, link, source, title)
|
||||||
|
for source, title, link in batch
|
||||||
|
]
|
||||||
|
|
||||||
|
for future in as_completed(futures):
|
||||||
|
try:
|
||||||
|
result = future.result(timeout=120) # 2 minute timeout
|
||||||
|
results.append(result)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error in pull_article: {e}")
|
||||||
|
errors.append(e)
|
||||||
|
|
||||||
|
# Add a small delay between batches to reduce system load
|
||||||
|
time.sleep(5)
|
||||||
|
|
||||||
|
return results, errors
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""
|
||||||
|
Main scraping loop.
|
||||||
|
"""
|
||||||
|
while True:
|
||||||
print("=========================================")
|
print("=========================================")
|
||||||
print("Starting new scraping iteration...")
|
print("Starting new scraping iteration...")
|
||||||
|
|
||||||
|
try:
|
||||||
# Pull the RSS feed sources from the JSON file
|
# Pull the RSS feed sources from the JSON file
|
||||||
rss_feed_sources = load_rss_feed_sources()
|
rss_feed_sources = load_rss_feed_sources()
|
||||||
|
|
||||||
@ -191,27 +296,26 @@ while True:
|
|||||||
|
|
||||||
# Randomize the order of the links to help with load balancing
|
# Randomize the order of the links to help with load balancing
|
||||||
import random
|
import random
|
||||||
|
|
||||||
random.shuffle(rss_feed_links)
|
random.shuffle(rss_feed_links)
|
||||||
|
|
||||||
link_list = [link for _, title, link in rss_feed_links]
|
print(f"Found {len(rss_feed_links)} articles to process")
|
||||||
source_list = [source for source, _, _ in rss_feed_links]
|
|
||||||
title_list = [title for _, title, link in rss_feed_links]
|
if not rss_feed_links:
|
||||||
|
print("No articles found, sleeping for 15 minutes")
|
||||||
|
time.sleep(15 * 60)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Process articles with better error handling and resource management
|
||||||
|
results, errors = safe_pull_articles(rss_feed_links)
|
||||||
|
|
||||||
with ProcessPoolExecutor() as executor:
|
|
||||||
futures = [executor.submit(pull_article, link, source, title) for link, source, title in zip(link_list, source_list, title_list)]
|
|
||||||
results = []
|
|
||||||
errors = []
|
|
||||||
for future in futures:
|
|
||||||
try:
|
|
||||||
results.append(future.result(timeout=60)) # seconds
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error in pull_article: {e}")
|
|
||||||
errors.append(e)
|
|
||||||
print(f"Attempted to Pull {len(results)} articles in parallel.")
|
print(f"Attempted to Pull {len(results)} articles in parallel.")
|
||||||
print(f"Encountered {len(errors)} errors during article pulling. " +\
|
print(
|
||||||
"Outputting errors to a local file.")
|
f"Encountered {len(errors)} errors during article pulling. "
|
||||||
|
+ "Outputting errors to a local file."
|
||||||
|
)
|
||||||
|
|
||||||
# Ouput errors to a local file
|
# Output errors to a local file
|
||||||
if errors:
|
if errors:
|
||||||
with open("errors.txt", "w", encoding="utf-8") as f:
|
with open("errors.txt", "w", encoding="utf-8") as f:
|
||||||
for error in errors:
|
for error in errors:
|
||||||
@ -224,6 +328,14 @@ while True:
|
|||||||
f.write(result + "\n")
|
f.write(result + "\n")
|
||||||
print("All articles pulled successfully.")
|
print("All articles pulled successfully.")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Major error in main loop: {e}")
|
||||||
|
# Continue to next iteration even if there's a major error
|
||||||
|
|
||||||
# Sleep for a while before the next iteration
|
# Sleep for a while before the next iteration
|
||||||
print("Sleeping for 15 minutes before the next iteration...")
|
print("Sleeping for 15 minutes before the next iteration...")
|
||||||
time.sleep(15 * 60) # Sleep for 15 minutes
|
time.sleep(15 * 60) # Sleep for 15 minutes
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user