refactor: split into StockDocs (core), stockdocs-scraper, stockdocs-mcp
Move the RSS scraper into its own repository and the MCP data server into its own repository. This repo keeps the article server, AI processor, and embedding service. Compose and pyproject trimmed accordingly.
This commit is contained in:
parent
8030ef8efe
commit
04f7365ac4
@ -1,104 +0,0 @@
|
||||
# MCPServer
|
||||
|
||||
A Flask-based server that provides an API for accessing and analyzing financial data, with integration for stock analysis and news processing.
|
||||
|
||||
## Overview
|
||||
|
||||
The MCPServer (Model Calling Protocol Server) is designed to provide an API interface for financial data processing including stock market analysis, news aggregation, and financial metrics calculation. It serves as the backend service for processing financial information and making it available through HTTP endpoints.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
MCPServer/
|
||||
├── app.py # Main Flask application
|
||||
├── config.py # Configuration settings
|
||||
├── requirements.txt # Python dependencies
|
||||
├── Dockerfile # Docker configuration
|
||||
├── README.md # This file
|
||||
└── api/ # API endpoints and handlers
|
||||
├── __init__.py
|
||||
├── stock_analysis.py # Stock analysis functions
|
||||
└── news_processing.py # News processing functions
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Stock Analysis**: Financial metrics calculation and market data processing
|
||||
- **News Integration**: APIs for retrieving and processing financial news
|
||||
- **Data Aggregation**: Consolidation of multiple data sources into unified responses
|
||||
- **RESTful API**: Clean HTTP interface for external services to consume data
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Stock Analysis
|
||||
- GET `/api/stock/metrics` - Get financial metrics for a stock
|
||||
- GET `/api/stock/history` - Get historical price data
|
||||
- POST `/api/stock/analyze` - Perform comprehensive stock analysis
|
||||
|
||||
### News Processing
|
||||
- GET `/api/news` - Retrieve news articles related to stocks
|
||||
- GET `/api/news/outlets` - List available news sources
|
||||
- POST `/api/news/process` - Process and categorize news content
|
||||
|
||||
### Data Management
|
||||
- GET `/api/data/refresh` - Refresh data from sources
|
||||
- GET `/api/status` - Server health check
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The server supports configuration through environment variables:
|
||||
|
||||
- `FLASK_ENV` - Set to 'development' or 'production' (default: 'development')
|
||||
- `DATABASE_URL` - URL for database connection (e.g., PostgreSQL)
|
||||
- `API_KEY` - API key for external services
|
||||
- `AI_SERVICE_API_KEY` - API key for authenticating with the centralized AI service at http://example.com:4000
|
||||
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
||||
|
||||
## Installation
|
||||
|
||||
1. Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
2. Set up environment variables (optional but recommended):
|
||||
```bash
|
||||
export FLASK_ENV=production
|
||||
export DATABASE_URL="postgresql://user:password@localhost/dbname"
|
||||
```
|
||||
|
||||
3. Run the server:
|
||||
```bash
|
||||
python app.py
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Get stock metrics:
|
||||
```bash
|
||||
curl "http://localhost:5000/api/stock/metrics?symbol=AAPL"
|
||||
```
|
||||
|
||||
### Get historical data:
|
||||
```bash
|
||||
curl "http://localhost:5000/api/stock/history?symbol=AAPL&days=30"
|
||||
```
|
||||
|
||||
### Process news for a stock:
|
||||
```bash
|
||||
curl -X POST "http://localhost:5000/api/news/process" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"symbol":"AAPL","articles":["/path/to/article1.txt","/path/to/article2.txt"]}'
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.6+
|
||||
- Flask 2.3.3
|
||||
- Additional dependencies listed in `requirements.txt`
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License.
|
||||
@ -1,20 +0,0 @@
|
||||
# Use an official Python runtime as a base image
|
||||
FROM python:3.12.3-slim
|
||||
|
||||
# Set the working directory in the container
|
||||
WORKDIR /app
|
||||
|
||||
# Install Dependencies
|
||||
COPY requirements.txt /app/
|
||||
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy the current directory contents into the container at /app
|
||||
COPY . /app
|
||||
|
||||
# Expose the port the app runs on
|
||||
EXPOSE 5005
|
||||
|
||||
# Run the Flask app
|
||||
CMD ["python", "server.py"]
|
||||
|
||||
@ -1,316 +0,0 @@
|
||||
{
|
||||
"openapi": "3.0.0",
|
||||
"info": {
|
||||
"title": "StockDocs",
|
||||
"version": "1.0.0"
|
||||
},
|
||||
"paths": {
|
||||
"/query": {
|
||||
"post": {
|
||||
"summary": "Query the vector database",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"question": { "type": "string" }
|
||||
},
|
||||
"required": ["question"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Query results",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"results": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"document": { "type": "string" },
|
||||
"score": { "type": "number" },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/articles/query": {
|
||||
"post": {
|
||||
"summary": "Query articles based on a question with diversity filtering",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"question": { "type": "string" },
|
||||
"max_results": { "type": "integer" }
|
||||
},
|
||||
"required": ["question"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Query results with diverse articles",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"results": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"document": { "type": "string" },
|
||||
"score": { "type": "number" },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"query": { "type": "string" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/articles/latest/{field}": {
|
||||
"get": {
|
||||
"summary": "Get latest articles about a specific field with diversity",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "field",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Latest diverse articles for the field",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"results": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"document": { "type": "string" },
|
||||
"score": { "type": "number" },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"field": { "type": "string" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/company/{company_name}/facts": {
|
||||
"get": {
|
||||
"summary": "Get facts about a specific company",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "company_name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Company facts",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"company": { "type": "string" },
|
||||
"facts": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/company/{company_name}/products": {
|
||||
"get": {
|
||||
"summary": "Get products information for a company",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "company_name",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Company products",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"company": { "type": "string" },
|
||||
"products": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": { "type": "string" },
|
||||
"release_date": { "type": "string" },
|
||||
"specifications": { "type": "string" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/company/facts/update": {
|
||||
"post": {
|
||||
"summary": "Update or add company facts",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"company_name": { "type": "string" },
|
||||
"facts": { "type": "object" }
|
||||
},
|
||||
"required": ["company_name", "facts"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Facts updated successfully",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"message": { "type": "string" },
|
||||
"company": { "type": "string" },
|
||||
"facts": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/health": {
|
||||
"get": {
|
||||
"summary": "Health check",
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/info": {
|
||||
"get": {
|
||||
"summary": "Service info",
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Info"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/facts": {
|
||||
"post": {
|
||||
"summary": "Get the best set of facts about a question",
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"question": { "type": "string" }
|
||||
},
|
||||
"required": ["question"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Facts and articles related to the question",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"question": { "type": "string" },
|
||||
"articles": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"document": { "type": "string" },
|
||||
"score": { "type": "number" },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"company_facts": { "type": "object" },
|
||||
"timestamp": { "type": "string", "format": "date-time" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,5 +0,0 @@
|
||||
home = /usr/bin
|
||||
include-system-site-packages = false
|
||||
version = 3.12.3
|
||||
executable = /usr/bin/python3.12
|
||||
command = /usr/bin/python3 -m venv /home/user/StockDocs/MCPServer
|
||||
@ -1,6 +0,0 @@
|
||||
Flask
|
||||
requests
|
||||
numpy
|
||||
pandas
|
||||
scikit-learn
|
||||
chromadb
|
||||
@ -1,412 +0,0 @@
|
||||
import chromadb
|
||||
from flask import Flask, request, jsonify, send_from_directory
|
||||
import os
|
||||
import requests
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
# Setup logging
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
# ChromaDB client setup
|
||||
try:
|
||||
client = chromadb.HttpClient(host="chromadb", port=8000)
|
||||
logger.info("Connected to ChromaDB successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to connect to ChromaDB: {e}")
|
||||
client = None
|
||||
|
||||
# Company facts database (in-memory for now, could be replaced with persistent storage)
|
||||
company_facts = {
|
||||
"Apple": {
|
||||
"products": [
|
||||
{"name": "iPhone", "release_date": "2007", "specifications": "Smartphone with iOS"},
|
||||
{"name": "MacBook", "release_date": "2006", "specifications": "Laptop with M1 chip"},
|
||||
{"name": "iPad", "release_date": "2010", "specifications": "Tablet with iOS"},
|
||||
{"name": "Apple Watch", "release_date": "2015", "specifications": "Smartwatch with watchOS"}
|
||||
],
|
||||
"founded": "1976",
|
||||
"ceo": "Tim Cook",
|
||||
"headquarters": "Cupertino, California",
|
||||
"ipo_year": "1980",
|
||||
"market_cap": "$2.8T (2026)",
|
||||
"key_executives": [
|
||||
{"name": "Tim Cook", "position": "CEO"},
|
||||
{"name": "Johny Srouji", "position": "CFO"},
|
||||
{"name": "Katherine Adams", "position": "Chief Design Officer"}
|
||||
],
|
||||
"business_segments": ["Consumer Electronics", "Software", "Services"]
|
||||
},
|
||||
"Microsoft": {
|
||||
"products": [
|
||||
{"name": "Windows", "release_date": "1985", "specifications": "Operating system"},
|
||||
{"name": "Office", "release_date": "1989", "specifications": "Productivity suite"},
|
||||
{"name": "Azure", "release_date": "2010", "specifications": "Cloud computing platform"},
|
||||
{"name": "Xbox", "release_date": "2001", "specifications": "Gaming console"}
|
||||
],
|
||||
"founded": "1975",
|
||||
"ceo": "Satya Nadella",
|
||||
"headquarters": "Redmond, Washington",
|
||||
"ipo_year": "1986",
|
||||
"market_cap": "$3.2T (2026)",
|
||||
"key_executives": [
|
||||
{"name": "Satya Nadella", "position": "CEO"},
|
||||
{"name": "Amy Hood", "position": "CFO"},
|
||||
{"name": "Kevin Scott", "position": "CTO"}
|
||||
],
|
||||
"business_segments": ["Software", "Cloud Services", "Gaming", "Productivity"]
|
||||
},
|
||||
"Google": {
|
||||
"products": [
|
||||
{"name": "Search Engine", "release_date": "1998", "specifications": "Web search platform"},
|
||||
{"name": "Android", "release_date": "2008", "specifications": "Mobile operating system"},
|
||||
{"name": "Gmail", "release_date": "2004", "specifications": "Email service"},
|
||||
{"name": "YouTube", "release_date": "2005", "specifications": "Video sharing platform"}
|
||||
],
|
||||
"founded": "1998",
|
||||
"ceo": "Sundar Pichai",
|
||||
"headquarters": "Mountain View, California",
|
||||
"ipo_year": "2004",
|
||||
"market_cap": "$1.7T (2026)",
|
||||
"key_executives": [
|
||||
{"name": "Sundar Pichai", "position": "CEO"},
|
||||
{"name": "Ruth Porat", "position": "CFO"},
|
||||
{"name": "Rajen S. Suri", "position": "Chief Technology Officer"}
|
||||
],
|
||||
"business_segments": ["Search", "Advertising", "Cloud", "Mobile"]
|
||||
},
|
||||
"Amazon": {
|
||||
"products": [
|
||||
{"name": "Amazon Web Services (AWS)", "release_date": "2006", "specifications": "Cloud computing platform"},
|
||||
{"name": "Kindle", "release_date": "2007", "specifications": "E-reader device"},
|
||||
{"name": "Alexa", "release_date": "2014", "specifications": "Voice assistant"},
|
||||
{"name": "Prime Video", "release_date": "2008", "specifications": "Streaming service"}
|
||||
],
|
||||
"founded": "1994",
|
||||
"ceo": "Andy Jassy",
|
||||
"headquarters": "Seattle, Washington",
|
||||
"ipo_year": "1997",
|
||||
"market_cap": "$1.5T (2026)",
|
||||
"key_executives": [
|
||||
{"name": "Andy Jassy", "position": "CEO"},
|
||||
{"name": "Brian T. Olsavsky", "position": "CFO"},
|
||||
{"name": "Wendy J. Smith", "position": "Chief Technology Officer"}
|
||||
],
|
||||
"business_segments": ["E-commerce", "Cloud Computing", "Digital Streaming", "Advertising"]
|
||||
}
|
||||
}
|
||||
|
||||
def get_embedding(text):
|
||||
"""
|
||||
Get embedding using the OpenAI-compatible server at http://example.com:4000
|
||||
"""
|
||||
try:
|
||||
# Using the OpenAI-compatible endpoint for embeddings
|
||||
headers = {
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
# Add API key if available
|
||||
api_key = os.getenv("AI_SERVICE_API_KEY")
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
|
||||
response = requests.post(
|
||||
"http://example.com:4000/v1/embeddings",
|
||||
json={
|
||||
"input": text,
|
||||
"model": "text-embedding-3-small" # or whatever model you're using
|
||||
},
|
||||
headers=headers,
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
embedding = response.json()['data'][0]['embedding']
|
||||
return embedding
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting embedding: {e}")
|
||||
return None
|
||||
|
||||
def get_diverse_articles(articles, max_diverse=5):
|
||||
"""
|
||||
Filter articles to ensure diversity in content, sources, and perspectives
|
||||
"""
|
||||
if len(articles) <= max_diverse:
|
||||
return articles
|
||||
|
||||
# More sophisticated diversity algorithm
|
||||
diverse_articles = []
|
||||
source_count = {}
|
||||
topic_count = {}
|
||||
|
||||
# First pass: try to get articles from different sources
|
||||
for article in articles:
|
||||
source = article.get('metadata', {}).get('source', 'unknown')
|
||||
topic = article.get('metadata', {}).get('topic', 'unknown')
|
||||
|
||||
# If we haven't reached max diversity and this source is new, add it
|
||||
if len(diverse_articles) < max_diverse and source not in source_count:
|
||||
diverse_articles.append(article)
|
||||
source_count[source] = 1
|
||||
topic_count[topic] = topic_count.get(topic, 0) + 1
|
||||
|
||||
# Second pass: fill remaining slots with different topics if possible
|
||||
if len(diverse_articles) < max_diverse:
|
||||
for article in articles:
|
||||
if len(diverse_articles) >= max_diverse:
|
||||
break
|
||||
source = article.get('metadata', {}).get('source', 'unknown')
|
||||
topic = article.get('metadata', {}).get('topic', 'unknown')
|
||||
|
||||
# Add article if it's from a different topic and we haven't seen too many from this topic
|
||||
if source not in source_count and topic_count.get(topic, 0) < 2:
|
||||
diverse_articles.append(article)
|
||||
source_count[source] = 1
|
||||
topic_count[topic] = topic_count.get(topic, 0) + 1
|
||||
|
||||
# If we still don't have enough, just return first few
|
||||
if len(diverse_articles) < max_diverse:
|
||||
return articles[:max_diverse]
|
||||
|
||||
return diverse_articles
|
||||
|
||||
def query_chroma(question, n_results=10):
|
||||
"""
|
||||
Query ChromaDB for articles related to the question
|
||||
"""
|
||||
if not client:
|
||||
return {"error": "ChromaDB connection failed"}
|
||||
|
||||
try:
|
||||
# Get embedding for the question
|
||||
query_embedding = get_embedding(question)
|
||||
if not query_embedding:
|
||||
return {"error": "Failed to get embedding"}
|
||||
|
||||
# Query the collection
|
||||
results = client.get_or_create_collection("news").query(
|
||||
query_embeddings=[query_embedding],
|
||||
n_results=n_results,
|
||||
)
|
||||
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.error(f"Error querying ChromaDB: {e}")
|
||||
return {"error": f"Query failed: {str(e)}"}
|
||||
|
||||
# New endpoints implementation based on the OpenAPI specification
|
||||
|
||||
@app.route("/query", methods=["POST"])
|
||||
def query_vector_database():
|
||||
"""Query the vector database"""
|
||||
data = request.get_json()
|
||||
question = data.get("question")
|
||||
|
||||
if not question:
|
||||
return jsonify({"error": "Missing 'question' in request body"}), 400
|
||||
|
||||
# Query ChromaDB for relevant articles
|
||||
results = query_chroma(question, n_results=10)
|
||||
|
||||
if "error" in results:
|
||||
return jsonify({"error": results["error"]}), 500
|
||||
|
||||
# Process results to create diverse article set
|
||||
mcp_results = []
|
||||
for doc, score, meta in zip(
|
||||
results.get("documents", [[]])[0],
|
||||
results.get("distances", [[]])[0],
|
||||
results.get("metadatas", [[]])[0]):
|
||||
mcp_results.append({
|
||||
"document": doc,
|
||||
"score": float(score),
|
||||
"metadata": meta
|
||||
})
|
||||
|
||||
# Apply diversity filtering
|
||||
diverse_results = get_diverse_articles(mcp_results, 5)
|
||||
|
||||
return jsonify({"results": diverse_results})
|
||||
|
||||
@app.route("/articles/query", methods=["POST"])
|
||||
def query_articles():
|
||||
"""Query articles based on a question with diversity filtering"""
|
||||
data = request.get_json()
|
||||
question = data.get("question")
|
||||
max_results = data.get("max_results", 5)
|
||||
|
||||
if not question:
|
||||
return jsonify({"error": "Missing 'question' in request body"}), 400
|
||||
|
||||
# Query ChromaDB for relevant articles
|
||||
results = query_chroma(question, n_results=max_results)
|
||||
|
||||
if "error" in results:
|
||||
return jsonify({"error": results["error"]}), 500
|
||||
|
||||
# Process results to create diverse article set
|
||||
mcp_results = []
|
||||
for doc, score, meta in zip(
|
||||
results.get("documents", [[]])[0],
|
||||
results.get("distances", [[]])[0],
|
||||
results.get("metadatas", [[]])[0]):
|
||||
mcp_results.append({
|
||||
"document": doc,
|
||||
"score": float(score),
|
||||
"metadata": meta
|
||||
})
|
||||
|
||||
# Apply diversity filtering
|
||||
diverse_results = get_diverse_articles(mcp_results, max_results)
|
||||
|
||||
return jsonify({
|
||||
"results": diverse_results,
|
||||
"query": question
|
||||
})
|
||||
|
||||
@app.route("/articles/latest/<field>", methods=["GET"])
|
||||
def get_latest_articles(field):
|
||||
"""Get latest articles about a specific field with diversity"""
|
||||
# This would typically query the database for latest articles about the field
|
||||
# For now, we'll return some sample data
|
||||
sample_articles = [
|
||||
{
|
||||
"document": f"Latest article about {field}",
|
||||
"score": 0.95,
|
||||
"metadata": {
|
||||
"source": "Sample Source",
|
||||
"topic": field,
|
||||
"date": "2026-01-31"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
return jsonify({
|
||||
"results": sample_articles,
|
||||
"field": field
|
||||
})
|
||||
|
||||
@app.route("/company/<company_name>/facts", methods=["GET"])
|
||||
def get_company_facts(company_name):
|
||||
"""Get facts about a specific company"""
|
||||
facts = company_facts.get(company_name, {})
|
||||
if not facts:
|
||||
return jsonify({"error": f"Company {company_name} not found"}), 404
|
||||
|
||||
return jsonify({
|
||||
"company": company_name,
|
||||
"facts": facts
|
||||
})
|
||||
|
||||
@app.route("/company/<company_name>/products", methods=["GET"])
|
||||
def get_company_products(company_name):
|
||||
"""Get products information for a company"""
|
||||
facts = company_facts.get(company_name, {})
|
||||
products = facts.get("products", [])
|
||||
|
||||
if not products:
|
||||
return jsonify({"error": f"No products found for company {company_name}"}), 404
|
||||
|
||||
return jsonify({
|
||||
"company": company_name,
|
||||
"products": products
|
||||
})
|
||||
|
||||
@app.route("/company/facts/update", methods=["POST"])
|
||||
def update_company_facts():
|
||||
"""Update or add company facts"""
|
||||
data = request.get_json()
|
||||
company_name = data.get("company_name")
|
||||
facts = data.get("facts")
|
||||
|
||||
if not company_name or not facts:
|
||||
return jsonify({"error": "Missing 'company_name' or 'facts' in request body"}), 400
|
||||
|
||||
# Update or add company facts
|
||||
company_facts[company_name] = facts
|
||||
|
||||
return jsonify({
|
||||
"message": "Facts updated successfully",
|
||||
"company": company_name,
|
||||
"facts": facts
|
||||
})
|
||||
|
||||
# Restore the original /facts endpoint
|
||||
@app.route("/facts", methods=["POST"])
|
||||
def get_facts():
|
||||
"""
|
||||
Get the best set of facts about a question
|
||||
"""
|
||||
data = request.get_json()
|
||||
question = data.get("question")
|
||||
|
||||
if not question:
|
||||
return jsonify({"error": "Missing 'question' in request body"}), 400
|
||||
|
||||
# Query ChromaDB for relevant articles
|
||||
results = query_chroma(question, n_results=10)
|
||||
|
||||
if "error" in results:
|
||||
return jsonify({"error": results["error"]}), 500
|
||||
|
||||
# Process results to create diverse article set
|
||||
mcp_results = []
|
||||
for doc, score, meta in zip(
|
||||
results.get("documents", [[]])[0],
|
||||
results.get("distances", [[]])[0],
|
||||
results.get("metadatas", [[]])[0]):
|
||||
mcp_results.append({
|
||||
"document": doc,
|
||||
"score": float(score),
|
||||
"metadata": meta
|
||||
})
|
||||
|
||||
# Apply diversity filtering
|
||||
diverse_results = get_diverse_articles(mcp_results, 5)
|
||||
|
||||
# Combine with company facts if question mentions a company
|
||||
company_facts_result = {}
|
||||
question_lower = question.lower()
|
||||
|
||||
# Check if question mentions any known company
|
||||
for company_name in company_facts.keys():
|
||||
if company_name.lower() in question_lower:
|
||||
company_facts_result = company_facts[company_name]
|
||||
break
|
||||
|
||||
# Return combined results
|
||||
response_data = {
|
||||
"question": question,
|
||||
"articles": diverse_results,
|
||||
"company_facts": company_facts_result,
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
return jsonify(response_data)
|
||||
|
||||
@app.route("/health", methods=["GET"])
|
||||
def health():
|
||||
"""Health check endpoint"""
|
||||
return jsonify({"status": "healthy", "timestamp": datetime.now().isoformat()})
|
||||
|
||||
@app.route("/info", methods=["GET"])
|
||||
def info():
|
||||
"""Service information endpoint"""
|
||||
return jsonify({
|
||||
"provider": "StockDoc",
|
||||
"service": "MCP Server",
|
||||
"version": "1.0.0",
|
||||
"collection": "news",
|
||||
"embedding_service": "http://example.com:4000"
|
||||
})
|
||||
|
||||
@app.route("/openapi.json", methods=["GET"])
|
||||
def openapi():
|
||||
"""Return OpenAPI specification"""
|
||||
return send_from_directory('.', 'openapi.json')
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run(host="0.0.0.0", port=5005, threaded=True, debug=True)
|
||||
156
README.md
156
README.md
@ -1,111 +1,89 @@
|
||||
# StockDocs
|
||||
|
||||
A comprehensive financial news analysis platform that combines web scraping, AI processing, and data embedding to provide actionable insights from financial news sources.
|
||||
Financial news analysis platform: serves a collected article corpus over HTTP, runs NLP sentiment/topic/entity analysis, and builds transformer embeddings for semantic search.
|
||||
|
||||
## Overview
|
||||
Part of the StockDocs project family:
|
||||
|
||||
StockDocs is a sophisticated system designed to collect, process, and analyze financial news from multiple sources. The platform consists of several interconnected components that work together to transform raw news content into valuable market intelligence.
|
||||
| Repo | What it is |
|
||||
|------|------------|
|
||||
| [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper) | RSS scraper — collects financial news from 60+ outlets (Reuters, Bloomberg, Forbes...) into the article corpus |
|
||||
| [stockdocs-mcp](https://git.jarianc.com/jarianc/stockdocs-mcp) | MCP server exposing the processed data to LLM clients |
|
||||
|
||||
This repository contains the processing core:
|
||||
|
||||
## Project Components
|
||||
|
||||
### 1. Scraper
|
||||
- Web scraping system for collecting financial news via RSS feeds
|
||||
- Organizes articles by news source in a structured directory hierarchy
|
||||
- Supports 60+ financial news outlets including Reuters, Bloomberg, Forbes, and more
|
||||
|
||||
### 2. Article Server
|
||||
- Flask-based HTTP server providing access to collected articles
|
||||
### 1. Article Server
|
||||
- Flask-based HTTP server providing access to the collected article corpus
|
||||
- Query articles by time range and news outlet filters
|
||||
- Retrieve full article content by file path
|
||||
- Exposes RESTful API for external applications
|
||||
|
||||
### 3. AI Processor
|
||||
### 2. AI Processor
|
||||
- Natural language processing engine for analyzing news content
|
||||
- Performs sentiment analysis, topic classification, and entity extraction
|
||||
- Generates actionable insights from financial articles
|
||||
- Generates structured facts from financial articles
|
||||
- Supports batch processing of large volumes of content
|
||||
- Prometheus metrics for pipeline observability
|
||||
|
||||
### 4. Embedding Service
|
||||
- Converts text content into numerical vector representations
|
||||
### 3. Embedding Service
|
||||
- Converts text content into numerical vector representations (ChromaDB)
|
||||
- Enables semantic similarity comparisons between articles
|
||||
- Supports various transformer-based models for high-quality embeddings
|
||||
- Provides caching mechanism to optimize performance
|
||||
|
||||
### 5. MCPServer
|
||||
- Market Capitalization Processor Server
|
||||
- Provides financial data processing and API interface
|
||||
- Integrates with stock analysis and news processing functions
|
||||
- Serves as backend service for external access to financial data
|
||||
- Transformer-based models for high-quality embeddings
|
||||
- Caching mechanism to avoid re-embedding processed articles
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
+--------------+ +--------------+ +-------------------+
|
||||
| Scraper | | Article | | AI |
|
||||
| (RSS Feeds) |--> | Server |--> | Processor |
|
||||
| | | | | |
|
||||
+--------------+ +--------------+ +-------------------+
|
||||
| |
|
||||
v v
|
||||
+--------------+ +-------------------+
|
||||
| Embedding | | MCPServer |
|
||||
| Service | | (Financial Data) |
|
||||
| | | |
|
||||
+--------------+ +-------------------+
|
||||
+---------------------+
|
||||
| Article corpus | <- populated by stockdocs-scraper
|
||||
+---------------------+
|
||||
|
|
||||
+------------+------------+
|
||||
v v
|
||||
+---------------+ +----------------+
|
||||
| Article Server| | AI Processor |
|
||||
| (Flask) | | (NLP) |
|
||||
+---------------+ +----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| Embedding |
|
||||
| Service |
|
||||
+----------------+
|
||||
|
|
||||
v
|
||||
+----------------+
|
||||
| ChromaDB |
|
||||
+----------------+
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-source News Collection**: Aggregates content from major financial news outlets
|
||||
- **Real-time Processing**: Automated scraping and analysis pipeline
|
||||
- **Advanced Analytics**: NLP-powered sentiment and topic analysis
|
||||
- **Semantic Search**: Vector-based similarity comparisons
|
||||
- **RESTful APIs**: Clean interfaces for integration with external systems
|
||||
- **Containerized Deployment**: Docker support for easy deployment
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.6+
|
||||
- Python 3.9+
|
||||
- Docker (for containerized deployment)
|
||||
- Internet connection for RSS feed access
|
||||
- An article corpus directory (produced by [stockdocs-scraper](https://git.jarianc.com/jarianc/stockdocs-scraper))
|
||||
|
||||
### Installation
|
||||
|
||||
1. Clone the repository:
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd StockDocs
|
||||
```
|
||||
|
||||
2. Set up each component:
|
||||
```bash
|
||||
# For each component, follow specific installation instructions
|
||||
cd scraper && pip install -r requirements.txt
|
||||
cd articleServer && pip install -r requirements.txt
|
||||
cd ai_processor && pip install -r requirements.txt
|
||||
cd embedding && pip install -r requirements.txt
|
||||
cd MCPServer && pip install -r requirements.txt
|
||||
cd ../ai_processor && pip install -r requirements.txt
|
||||
cd ../embedding && pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. Configure environment variables as needed for each component
|
||||
### Running
|
||||
|
||||
4. Run individual services:
|
||||
```bash
|
||||
python scraper/scraper.py # Start scraping
|
||||
python articleServer/run_server.py # Start article server
|
||||
python ai_processor/app.py # Start AI processor
|
||||
python embedding/app.py # Start embedding service
|
||||
python MCPServer/app.py # Start MCP server
|
||||
python articleServer/run_server.py # article server on :5008
|
||||
python ai_processor/main.py # NLP pipeline
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Data Collection
|
||||
The scraper component automatically collects news from configured RSS feeds and stores articles in structured directories.
|
||||
Query the article server:
|
||||
|
||||
### API Access
|
||||
Use the article server's RESTful APIs to access collected content:
|
||||
```bash
|
||||
# Get recent articles
|
||||
curl "http://localhost:5008/articles?time_range=hour"
|
||||
@ -114,48 +92,10 @@ curl "http://localhost:5008/articles?time_range=hour"
|
||||
curl "http://localhost:5008/article/content?path=/path/to/article.txt"
|
||||
```
|
||||
|
||||
### Analysis
|
||||
The AI processor and embedding service provide advanced analysis capabilities through their respective APIs.
|
||||
|
||||
## Deployment
|
||||
|
||||
Each component can be run independently or containerized using the provided Dockerfiles:
|
||||
```bash
|
||||
# Build and run each component in Docker
|
||||
docker build -t stockdocs-scraper ./scraper
|
||||
docker run -p 5000:5000 stockdocs-scraper
|
||||
|
||||
docker build -t stockdocs-article-server ./articleServer
|
||||
docker run -p 5008:5008 stockdocs-article-server
|
||||
|
||||
# Continue for other components...
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -r scraper/requirements_clean.txt pytest
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
Unit tests cover the scraper's article-processing cache: load/save
|
||||
roundtrips, processing status tracking, and progress computation.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.9+
|
||||
- Flask 2.3.3
|
||||
- Various NLP and ML libraries
|
||||
- Docker (for containerized deployment)
|
||||
All three services containerize with the included Dockerfiles; `docker-compose.yml` wires them onto a shared network with a shared articles volume.
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please read our contribution guidelines before submitting pull requests.
|
||||
|
||||
## Support
|
||||
|
||||
For support, please open an issue on the GitHub repository.
|
||||
This project is licensed under the MIT License — see the LICENSE file for details.
|
||||
|
||||
@ -1,15 +1,4 @@
|
||||
services:
|
||||
flask-app:
|
||||
build: ./MCPServer
|
||||
platform: linux/amd64
|
||||
container_name: stockdocs-mcp
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- ainetwork
|
||||
ports:
|
||||
- "5005:5005"
|
||||
environment:
|
||||
- FLASK_ENV=development
|
||||
article-server:
|
||||
build: ./articleServer
|
||||
platform: linux/amd64
|
||||
@ -20,7 +9,7 @@ services:
|
||||
ports:
|
||||
- "5008:5008"
|
||||
volumes:
|
||||
- /home/user/StockDocs/scraper/articles:/app/articles
|
||||
- ./articles:/app/articles
|
||||
environment:
|
||||
- ARTICLE_DIR=/app/articles
|
||||
ai_processor:
|
||||
@ -31,7 +20,7 @@ services:
|
||||
networks:
|
||||
- ainetwork
|
||||
volumes:
|
||||
- ./scraper/articles:/app/articles
|
||||
- ./articles:/app/articles
|
||||
- ./ai_processor/output:/app/output
|
||||
environment:
|
||||
- AI_SERVICE_URL=http://example.com:4000
|
||||
@ -44,7 +33,7 @@ services:
|
||||
networks:
|
||||
- ainetwork
|
||||
volumes:
|
||||
- ./scraper/articles:/scraper/articles
|
||||
- ./articles:/scraper/articles
|
||||
- ./embedding/logs:/app/logs
|
||||
environment:
|
||||
- CHROMADB_HOST=example.com
|
||||
|
||||
@ -1,28 +1,23 @@
|
||||
[project]
|
||||
name = "stockdocs"
|
||||
version = "1.0.0"
|
||||
description = "Financial news analysis platform: RSS scraping, AI processing, embeddings, and MCP server"
|
||||
description = "Financial news analysis platform: article server, AI fact processing, and semantic embeddings"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9"
|
||||
license = { text = "MIT" }
|
||||
authors = [{ name = "Jarian Cottingham", email = "jarianc@proton.me" }]
|
||||
keywords = ["finance", "news", "scraping", "nlp", "embeddings", "mcp"]
|
||||
keywords = ["finance", "news", "nlp", "embeddings", "chromadb"]
|
||||
classifiers = [
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
]
|
||||
dependencies = [
|
||||
"feedparser>=6.0,<7.0",
|
||||
"flask>=2.3,<3.0",
|
||||
"requests>=2.31,<3.0",
|
||||
"beautifulsoup4>=4.12,<5.0",
|
||||
"lxml>=5.0",
|
||||
"nltk>=3.8",
|
||||
"newspaper4k>=0.2.8,<0.3",
|
||||
"selenium>=4.15",
|
||||
"pandas>=2.0",
|
||||
"pyyaml>=6.0",
|
||||
"tqdm>=4.64",
|
||||
"tldextract>=5.0",
|
||||
"numpy>=1.24",
|
||||
"chromadb>=0.4",
|
||||
"prometheus-client>=0.19",
|
||||
"python-dotenv>=1.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@ -43,6 +38,3 @@ exclude = [".git", "articles", "ai_processor/ai_processor"]
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "W"]
|
||||
ignore = ["E501"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
3
scraper/.gitignore
vendored
3
scraper/.gitignore
vendored
@ -1,3 +0,0 @@
|
||||
errors.txt
|
||||
results.txt
|
||||
venv
|
||||
@ -1,89 +0,0 @@
|
||||
# Scraper
|
||||
|
||||
A Python-based web scraping system designed to collect financial news and articles from various sources using RSS feeds and automated scraping techniques.
|
||||
|
||||
## Overview
|
||||
|
||||
The Scraper component is responsible for collecting financial news and articles from multiple sources including major news outlets, financial publications, and market analysis services. It uses RSS feeds to gather content and stores the articles in a structured directory hierarchy for easy access by other components of the system.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
scraper/
|
||||
├── rss_feeds.json # Configuration file with RSS feed URLs
|
||||
├── scraper.py # Main scraping logic
|
||||
├── requirements.txt # Python dependencies
|
||||
├── dockerfile # Docker configuration
|
||||
├── dockerfile-selenium # Dockerfile for selenium-based scraping
|
||||
├── articles/ # Directory where scraped articles are stored
|
||||
│ ├── Reuters – Business News/
|
||||
│ │ ├── article1.txt
|
||||
│ │ └── ...
|
||||
│ ├── Associated Press – Business/
|
||||
│ │ ├── article1.txt
|
||||
│ │ └── ...
|
||||
│ └── ...
|
||||
├── processed_articles_cache.json # Cache of already processed articles
|
||||
└── pyvenv.cfg # Python virtual environment configuration
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **RSS Feed Integration**: Supports multiple financial news sources through RSS feeds
|
||||
- **Automated Scraping**: Regularly fetches and processes new articles from configured feeds
|
||||
- **Structured Storage**: Organizes articles in a directory structure by news outlet
|
||||
- **Duplicate Detection**: Prevents re-processing of already collected articles
|
||||
- **Caching Mechanism**: Maintains a cache of processed articles to optimize performance
|
||||
|
||||
## RSS Feed Sources
|
||||
|
||||
The scraper supports 60+ news outlets including:
|
||||
|
||||
- Reuters – Business News
|
||||
- Associated Press – Business
|
||||
- Financial Times
|
||||
- Forbes – Real-Time
|
||||
- Wall Street Journal – U.S. Business
|
||||
- Bloomberg – Surveillance Podcast
|
||||
- CNN Money
|
||||
- BBC News – Business
|
||||
- And many more...
|
||||
|
||||
## Usage
|
||||
|
||||
### Running the Scraper
|
||||
|
||||
```bash
|
||||
python scraper.py
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
The scraper can be configured by modifying `rss_feeds.json` to:
|
||||
- Add new news sources
|
||||
- Update existing RSS feed URLs
|
||||
- Remove sources that are no longer active
|
||||
|
||||
### Article Storage
|
||||
|
||||
Articles are stored in `articles/` directory with the following structure:
|
||||
|
||||
```
|
||||
articles/
|
||||
└── <News Outlet Name>/
|
||||
├── article1.txt
|
||||
├── article2.txt
|
||||
└── ...
|
||||
```
|
||||
|
||||
Where each article file contains the full text content of that news article.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.6+
|
||||
- Selenium WebDriver (for certain scraping operations)
|
||||
- Additional dependencies listed in `requirements.txt`
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License.
|
||||
@ -1,488 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Cron-based scraper for downloading articles from RSS feeds.
|
||||
This version replaces the infinite while loop with a single execution
|
||||
that can be scheduled via cron job.
|
||||
"""
|
||||
|
||||
import newspaper
|
||||
import json
|
||||
import feedparser
|
||||
import time
|
||||
import os
|
||||
import requests
|
||||
import logging
|
||||
import random
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.firefox.options import Options as FirefoxOptions
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
import nltk
|
||||
from nltk.downloader import Downloader
|
||||
|
||||
# Setup logging with timestamps
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Rotating User-Agents to bypass bot detection (Reuters, etc.)
|
||||
USER_AGENTS = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
]
|
||||
|
||||
def get_random_ua():
|
||||
return random.choice(USER_AGENTS)
|
||||
|
||||
# Robust file path handling - try multiple locations
|
||||
def get_feed_file_path():
|
||||
"""Get the RSS feed file path, trying multiple locations."""
|
||||
possible_paths = [
|
||||
"./rss_feeds.json", # Current directory
|
||||
"../rss_feeds.json", # Parent directory
|
||||
"/app/rss_feeds.json", # Docker path
|
||||
"./scraper/rss_feeds.json" # Scraper subdirectory
|
||||
]
|
||||
|
||||
for path in possible_paths:
|
||||
if os.path.exists(path):
|
||||
print(f"Found feed file at: {path}")
|
||||
return path
|
||||
|
||||
# If no file found, exit the program
|
||||
print("Error: RSS feed file not found in any expected location")
|
||||
print("Exiting program...")
|
||||
exit(1)
|
||||
|
||||
# Get the feed file path
|
||||
FEED_FILE = get_feed_file_path()
|
||||
|
||||
# Ensure necessary NLTK resources are downloaded
|
||||
d = Downloader()
|
||||
if not d.is_installed("punkt_tab"):
|
||||
nltk.download("punkt_tab")
|
||||
|
||||
articles = []
|
||||
|
||||
|
||||
def load_rss_feed_sources(feed_file=FEED_FILE):
|
||||
"""
|
||||
Loads the RSS feed sources from a JSON file.
|
||||
"""
|
||||
logger.info(f"Loading RSS feed sources from {feed_file}...")
|
||||
|
||||
# Debug: Print current working directory
|
||||
logger.debug(f"Current working directory: {os.getcwd()}")
|
||||
|
||||
try:
|
||||
with open(feed_file, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
logger.info(f"Successfully loaded {feed_file}")
|
||||
logger.debug(f"Data type: {type(data)}")
|
||||
if isinstance(data, dict) and "rss_feeds" in data:
|
||||
logger.info(f"Found rss_feeds section with {len(data['rss_feeds'])} sources")
|
||||
return data
|
||||
else:
|
||||
logger.warning(f"Unexpected data structure. Data keys: {list(data.keys()) if isinstance(data, dict) else 'Not a dict'}")
|
||||
return {}
|
||||
except FileNotFoundError:
|
||||
logger.error(f"{feed_file} not found, returning empty dict.")
|
||||
return {}
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"Error decoding {feed_file}: {e}, returning empty dict.")
|
||||
return {}
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error loading {feed_file}: {e}")
|
||||
return {}
|
||||
|
||||
|
||||
def mine_all_articles(rss_feed_sources, limit=None):
|
||||
"""
|
||||
Mines all articles from the given RSS feed sources.
|
||||
Returns a list of (site, title, link) tuples.
|
||||
"""
|
||||
all_links = []
|
||||
|
||||
# Check if rss_feed_sources is a valid dict with rss_feeds key
|
||||
if not isinstance(rss_feed_sources, dict):
|
||||
logger.warning(f"rss_feed_sources is not a dict, it's {type(rss_feed_sources)}")
|
||||
return all_links
|
||||
|
||||
if "rss_feeds" not in rss_feed_sources:
|
||||
logger.warning("rss_feeds key not found in rss_feed_sources")
|
||||
return all_links
|
||||
|
||||
sources = rss_feed_sources["rss_feeds"]
|
||||
|
||||
for site, data in sources.items():
|
||||
logger.info(f"Parsing RSS feed: {data['rss_url']}")
|
||||
try:
|
||||
feed = feedparser.parse(data["rss_url"])
|
||||
feed_entries = feed.entries[:limit] if limit else feed.entries
|
||||
|
||||
for entry in feed_entries:
|
||||
if "link" in entry and "title" in entry:
|
||||
all_links.append((site, entry.title, entry.link))
|
||||
except Exception as e:
|
||||
logger.error(f"Error parsing RSS feed: {site} Error: {str(e)}")
|
||||
return all_links
|
||||
|
||||
|
||||
def generate_filename_from_url(url):
|
||||
"""
|
||||
Generates a filename from the given URL by replacing slashes with underscores.
|
||||
"""
|
||||
# Use only the last part of the URL or replace slashes
|
||||
return url.replace("https://", "").replace("http://", "").replace("/", "_")
|
||||
|
||||
|
||||
def generate_safe_filename(name):
|
||||
# Remove/replace characters not allowed in filenames
|
||||
import re
|
||||
|
||||
safe = re.sub(r'[\\/*?:"<>|]', "_", name)
|
||||
return safe
|
||||
|
||||
|
||||
def save_article_to_file(article, filename, source="Unfiltered"):
|
||||
"""
|
||||
Saves the given article text to a file with the specified filename.
|
||||
"""
|
||||
# articles dir should already be there
|
||||
# os.makedirs("articles", exist_ok=True)
|
||||
|
||||
outputDir = "articles/" + source
|
||||
os.makedirs(outputDir, exist_ok=True) if source else None
|
||||
|
||||
# Sanitize filename: use only the last part of the URL or replace slashes
|
||||
safe_filename = generate_filename_from_url(filename)
|
||||
file_path = os.path.join(outputDir, safe_filename)
|
||||
# Save the source as the first line in the file for later retrieval
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
f.write(f"SOURCE:{source}\n")
|
||||
f.write(article)
|
||||
|
||||
# Only log when a new file is actually created (not cached)
|
||||
logger.info(f"New article saved: {safe_filename} from {source}")
|
||||
|
||||
|
||||
def get_article_with_selenium(url):
|
||||
"""
|
||||
Gets article text using Selenium Firefox driver with proper error handling,
|
||||
cleanup, and bot-detection evasion.
|
||||
"""
|
||||
driver = None
|
||||
try:
|
||||
# Configure Firefox options with bot-detection evasion
|
||||
options = FirefoxOptions()
|
||||
options.add_argument("--headless")
|
||||
options.set_preference("dom.ipc.processCount", 1)
|
||||
options.set_preference("general.useragent.override", get_random_ua())
|
||||
options.set_preference("permissions.default.image", 2)
|
||||
options.set_preference("dom.webnotifications.enabled", False)
|
||||
|
||||
# Initialize driver with timeout
|
||||
driver = webdriver.Firefox(options=options)
|
||||
driver.set_page_load_timeout(30)
|
||||
|
||||
# Navigate to URL
|
||||
driver.get(url)
|
||||
|
||||
# Wait for page to load (explicit wait instead of sleep)
|
||||
try:
|
||||
WebDriverWait(driver, 15).until(
|
||||
EC.presence_of_element_located((By.TAG_NAME, "body"))
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
time.sleep(random.uniform(1, 3))
|
||||
|
||||
html = driver.page_source
|
||||
|
||||
# Parse with Newspaper4k
|
||||
article = newspaper.article(url, input_html=html, language="en")
|
||||
article.nlp()
|
||||
logger.info(f"Successfully extracted article with Selenium from {url}")
|
||||
return article.text
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Selenium failed for {url}: {str(e)}")
|
||||
return ""
|
||||
finally:
|
||||
if driver:
|
||||
try:
|
||||
driver.quit()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def get_article_with_playwright(url):
|
||||
"""
|
||||
Gets article text using Playwright with proper bot-detection evasion.
|
||||
"""
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True, timeout=30000)
|
||||
context = browser.new_context(
|
||||
user_agent=get_random_ua(),
|
||||
viewport={"width": 1920, "height": 1080},
|
||||
locale="en-US",
|
||||
timezone_id="America/New_York",
|
||||
)
|
||||
page = context.new_page()
|
||||
|
||||
page.set_extra_http_headers({
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Connection": "keep-alive",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
})
|
||||
|
||||
page.goto(url, wait_until="domcontentloaded", timeout=30000)
|
||||
time.sleep(random.uniform(2, 4))
|
||||
|
||||
html = page.content()
|
||||
context.close()
|
||||
browser.close()
|
||||
|
||||
article = newspaper.article(url, input_html=html, language="en")
|
||||
article.nlp()
|
||||
logger.info(f"Successfully extracted article with Playwright from {url}")
|
||||
return article.text
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Playwright failed for {url}: {str(e)}")
|
||||
return ""
|
||||
|
||||
|
||||
def pull_article(link, source, title=None, save_to_file=True):
|
||||
"""
|
||||
Pulls an article from a given link with fallback mechanisms.
|
||||
"""
|
||||
filename = title if title else link
|
||||
safe_filename = generate_filename_from_url(filename)
|
||||
|
||||
# Check if already cached
|
||||
if os.path.exists(os.path.join("articles", source, safe_filename)):
|
||||
logger.info(f"Article already cached: {filename}")
|
||||
with open(
|
||||
os.path.join("articles", source, safe_filename), "r", encoding="utf-8"
|
||||
) as f:
|
||||
return f.read()
|
||||
|
||||
# Random delay before fetching to avoid rate-limiting / bot detection
|
||||
time.sleep(random.uniform(0.5, 2))
|
||||
|
||||
text = ""
|
||||
|
||||
try:
|
||||
# Try newspaper4k first with proper User-Agent to bypass bot detection
|
||||
ua = get_random_ua()
|
||||
article = newspaper.article(link, browser_user_agent=ua)
|
||||
article.download()
|
||||
article.parse()
|
||||
text = article.text
|
||||
|
||||
if not text or len(text) < 200:
|
||||
raise ValueError(
|
||||
"\tArticle text too short, falling back to Playwright/Selenium."
|
||||
)
|
||||
logger.info(f"Successfully pulled article with newspaper4k from {link}")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"newspaper4k extraction failed for {link}: {e}, falling back to Playwright."
|
||||
)
|
||||
|
||||
try:
|
||||
text = get_article_with_playwright(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Playwright")
|
||||
|
||||
if not text or len(text) < 200:
|
||||
logger.warning("Playwright article too short, falling back to Selenium.")
|
||||
# Fallback to Selenium with better error handling
|
||||
text = get_article_with_selenium(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Selenium")
|
||||
except Exception as e:
|
||||
logger.error(f"Playwright failed for {link}: {e}")
|
||||
|
||||
# Fallback to Selenium
|
||||
try:
|
||||
text = get_article_with_selenium(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Selenium")
|
||||
except Exception as e:
|
||||
logger.error(f"Selenium failed for {link}: {e}")
|
||||
return ""
|
||||
|
||||
if save_to_file:
|
||||
save_article_to_file(text, filename, source)
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def safe_pull_articles(article_list):
|
||||
"""
|
||||
Safely pull articles with improved error handling and reduced parallelism.
|
||||
"""
|
||||
results = []
|
||||
errors = []
|
||||
|
||||
# Process in smaller batches to reduce resource strain
|
||||
batch_size = 5
|
||||
|
||||
for i in range(0, len(article_list), batch_size):
|
||||
batch = article_list[i : i + batch_size]
|
||||
logger.info(f"Processing batch {i // batch_size + 1} with {len(batch)} articles")
|
||||
|
||||
# Use ThreadPoolExecutor instead of ProcessPoolExecutor to avoid
|
||||
# process termination issues with browser automation
|
||||
with ThreadPoolExecutor(max_workers=3) as executor: # Reduced workers
|
||||
futures = [
|
||||
executor.submit(pull_article, link, source, title)
|
||||
for source, title, link in batch
|
||||
]
|
||||
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
result = future.result(timeout=120) # 2 minute timeout
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
logger.error(f"Error in pull_article: {e}")
|
||||
errors.append(e)
|
||||
|
||||
# Add a small delay between batches to reduce system load
|
||||
time.sleep(5)
|
||||
|
||||
return results, errors
|
||||
|
||||
|
||||
def gather_new_articles():
|
||||
"""
|
||||
Gather list of all newly downloaded articles and format them for webhook.
|
||||
"""
|
||||
new_articles = []
|
||||
|
||||
# Walk through all article directories
|
||||
for root, dirs, files in os.walk("articles"):
|
||||
for file in files:
|
||||
if file != "processed_articles_cache.json": # Skip cache file
|
||||
# Get the full file path
|
||||
file_path = os.path.join(root, file)
|
||||
|
||||
# Get the outlet name from the directory path
|
||||
outlet = os.path.basename(root)
|
||||
|
||||
# Create the relative path for the article
|
||||
relative_path = os.path.relpath(file_path, "scraper")
|
||||
|
||||
# Create article data structure
|
||||
article_data = {
|
||||
"created_at": time.strftime("%Y-%m-%dT%H:%M:%S.%f", time.localtime(os.path.getctime(file_path))),
|
||||
"name": file,
|
||||
"outlet": outlet,
|
||||
"path": f"../{relative_path}"
|
||||
}
|
||||
|
||||
new_articles.append(article_data)
|
||||
|
||||
return new_articles
|
||||
|
||||
|
||||
def send_to_webhook(articles):
|
||||
"""
|
||||
Send list of articles to the webhook URL.
|
||||
"""
|
||||
webhook_url = "http://agents.example.com/webhook/49c5b169-c68c-4f8c-90c2-0fcca6e2d387"
|
||||
headers = {
|
||||
"StockDocsN8NAuthToken": "ganvT4gsgRjWpGE8FMw9uCzFjZrTx8RZCoVm2Dh7skbZecov"
|
||||
}
|
||||
|
||||
try:
|
||||
response = requests.post(webhook_url, json=articles, headers=headers, timeout=30)
|
||||
if response.status_code == 200:
|
||||
print(f"Successfully sent {len(articles)} articles to webhook")
|
||||
else:
|
||||
print(f"Webhook request failed with status code: {response.status_code}")
|
||||
print(f"Response: {response.text}")
|
||||
except Exception as e:
|
||||
print(f"Error sending to webhook: {e}")
|
||||
|
||||
|
||||
def main():
|
||||
"""
|
||||
Main scraping function for cron execution.
|
||||
This replaces the infinite while loop with a single execution.
|
||||
"""
|
||||
logger.info("=========================================")
|
||||
logger.info("Starting new scraping iteration...")
|
||||
|
||||
try:
|
||||
# Pull the RSS feed sources from the JSON file
|
||||
rss_feed_sources = load_rss_feed_sources()
|
||||
|
||||
# Mine all articles from the RSS feed sources
|
||||
rss_feed_links = mine_all_articles(rss_feed_sources)
|
||||
|
||||
# Randomize the order of the links to help with load balancing
|
||||
import random
|
||||
|
||||
random.shuffle(rss_feed_links)
|
||||
|
||||
logger.info(f"Found {len(rss_feed_links)} articles to process")
|
||||
|
||||
if not rss_feed_links:
|
||||
logger.info("No articles found")
|
||||
return
|
||||
|
||||
# Process articles with better error handling and resource management
|
||||
results, errors = safe_pull_articles(rss_feed_links)
|
||||
|
||||
logger.info(f"Attempted to Pull {len(results)} articles in parallel.")
|
||||
logger.info(
|
||||
f"Encountered {len(errors)} errors during article pulling. "
|
||||
+ "Outputting errors to a local file."
|
||||
)
|
||||
|
||||
# Output errors to a local file
|
||||
if errors:
|
||||
with open("errors.txt", "w", encoding="utf-8") as f:
|
||||
for error in errors:
|
||||
f.write(str(error) + "\n")
|
||||
logger.info("Errors logged to errors.txt")
|
||||
|
||||
# Print all results to a log file
|
||||
with open("results.txt", "w", encoding="utf-8") as f:
|
||||
for result in results:
|
||||
f.write(result + "\n")
|
||||
logger.info("All articles pulled successfully.")
|
||||
|
||||
# Gather and send new articles to webhook
|
||||
new_articles = gather_new_articles()
|
||||
if new_articles:
|
||||
send_to_webhook(new_articles)
|
||||
else:
|
||||
logger.info("No new articles to send to webhook")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Major error in main execution: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise # Re-raise to ensure the script exits with error code
|
||||
|
||||
logger.info("Scraping completed successfully.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,20 +0,0 @@
|
||||
# --- Dockerfile.local ---
|
||||
FROM python:3.13.5
|
||||
|
||||
# Firefox + GeckoDriver
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
firefox-esr wget ca-certificates gnupg2 \
|
||||
&& GECKO=v0.36.0 && \
|
||||
wget -qO- "https://github.com/mozilla/geckodriver/releases/download/${GECKO}/geckodriver-${GECKO}-linux64.tar.gz" \
|
||||
| tar -xz -C /usr/local/bin geckodriver \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN playwright install
|
||||
RUN playwright install-deps
|
||||
RUN python3 -m nltk.downloader punkt_tab # Download NLTK data and bake into image
|
||||
COPY . .
|
||||
|
||||
CMD ["python3", "scraper.py"]
|
||||
@ -1,22 +0,0 @@
|
||||
# Use an official Selenium Firefox standalone as a base image
|
||||
FROM selenium/standalone-firefox:latest
|
||||
|
||||
USER root
|
||||
# Set the working directory in the container
|
||||
WORKDIR /app
|
||||
|
||||
# Install Python dependencies
|
||||
COPY requirements.txt /app/
|
||||
RUN pip install --break-system-packages --no-cache-dir -r requirements.txt && \
|
||||
playwright install
|
||||
|
||||
# Copy the current directory contents into the container at /app
|
||||
COPY . /app
|
||||
RUN mkdir -p /app/articles && \
|
||||
chown seluser:seluser /app/articles && \
|
||||
chmod 755 /app/articles
|
||||
|
||||
USER seluser
|
||||
# Run the Flask app
|
||||
CMD ["python3", "scraper.py"]
|
||||
|
||||
@ -1,43 +0,0 @@
|
||||
attrs
|
||||
beautifulsoup4
|
||||
certifi
|
||||
charset-normalizer
|
||||
click
|
||||
dnspython
|
||||
feedparser
|
||||
filelock
|
||||
gnews
|
||||
greenlet
|
||||
h11
|
||||
idna
|
||||
joblib
|
||||
lxml
|
||||
lxml-html-clean
|
||||
newspaper4k
|
||||
nltk
|
||||
numpy
|
||||
outcome
|
||||
pandas
|
||||
pillow
|
||||
playwright
|
||||
pyee
|
||||
pysocks
|
||||
python-dateutil
|
||||
pytz
|
||||
pyyaml
|
||||
regex
|
||||
requests
|
||||
requests-file
|
||||
selenium
|
||||
sgmllib3k
|
||||
six
|
||||
sniffio
|
||||
sortedcontainers
|
||||
soupsieve
|
||||
tldextract
|
||||
tqdm
|
||||
trio
|
||||
trio-websocket
|
||||
tzdata
|
||||
websocket-client
|
||||
wsproto
|
||||
@ -1,43 +0,0 @@
|
||||
attrs
|
||||
beautifulsoup4
|
||||
certifi
|
||||
charset-normalizer
|
||||
click
|
||||
dnspython
|
||||
feedparser
|
||||
filelock
|
||||
gnews
|
||||
greenlet
|
||||
h11
|
||||
idna
|
||||
joblib
|
||||
lxml
|
||||
lxml-html-clean
|
||||
newspaper4k
|
||||
nltk
|
||||
numpy
|
||||
outcome
|
||||
pandas
|
||||
pillow
|
||||
playwright
|
||||
pyee
|
||||
pysocks
|
||||
python-dateutil
|
||||
pytz
|
||||
pyyaml
|
||||
regex
|
||||
requests
|
||||
requests-file
|
||||
selenium
|
||||
sgmllib3k
|
||||
six
|
||||
sniffio
|
||||
sortedcontainers
|
||||
soupsieve
|
||||
tldextract
|
||||
tqdm
|
||||
trio
|
||||
trio-websocket
|
||||
tzdata
|
||||
websocket-client
|
||||
wsproto
|
||||
@ -1,196 +0,0 @@
|
||||
{
|
||||
"rss_feeds": {
|
||||
"Reuters – Business News": {
|
||||
"source_website": "reuters.com",
|
||||
"rss_url": "https://www.reutersagency.com/feed/"
|
||||
},
|
||||
"Associated Press – Business": {
|
||||
"source_website": "apnews.com",
|
||||
"rss_url": "https://rsshub.app/apnews/topics/apf-topnews"
|
||||
},
|
||||
"Financial Times": {
|
||||
"source_website": "ft.com",
|
||||
"rss_url": "https://www.ft.com/rss/home"
|
||||
},
|
||||
"Fortune – Top Stories": {
|
||||
"source_website": "fortune.com",
|
||||
"rss_url": "https://fortune.com/feed/fortune-feeds/?id=3230629"
|
||||
},
|
||||
"Seeking Alpha – Market News": {
|
||||
"source_website": "seekingalpha.com",
|
||||
"rss_url": "https://seekingalpha.com/feed.xml"
|
||||
},
|
||||
"The Motley Fool – Stock News & Analysis": {
|
||||
"source_website": "fool.com",
|
||||
"rss_url": "https://www.fool.com/a/feeds/partner/googlechromefollow?apikey=5e092c1f-c5f9-4428-9219-908a47d2e2de"
|
||||
},
|
||||
"Business Standard – Latest News": {
|
||||
"source_website": "business-standard.com",
|
||||
"rss_url": "https://www.business-standard.com/rss/latest.rss"
|
||||
},
|
||||
"TheStreet – Full Articles": {
|
||||
"source_website": "thestreet.com",
|
||||
"rss_url": "https://www.thestreet.com/.rss/full"
|
||||
},
|
||||
"Benzinga – Financial News": {
|
||||
"source_website": "benzinga.com",
|
||||
"rss_url": "https://feeds.benzinga.com/benzinga"
|
||||
},
|
||||
"MarketBeat – Market News": {
|
||||
"source_website": "marketbeat.com",
|
||||
"rss_url": "https://www.marketbeat.com/feed/"
|
||||
},
|
||||
"Money (Time) – Personal Finance": {
|
||||
"source_website": "money.com",
|
||||
"rss_url": "https://money.com/money/feed/"
|
||||
},
|
||||
"Global Finance Magazine": {
|
||||
"source_website": "gfmag.com",
|
||||
"rss_url": "https://www.gfmag.com/feed"
|
||||
},
|
||||
"Financial Samurai": {
|
||||
"source_website": "financialsamurai.com",
|
||||
"rss_url": "https://www.financialsamurai.com/feed/"
|
||||
},
|
||||
"MoneyWeek": {
|
||||
"source_website": "moneyweek.com",
|
||||
"rss_url": "https://moneyweek.com/feed/all"
|
||||
},
|
||||
"Finance Monthly": {
|
||||
"source_website": "finance-monthly.com",
|
||||
"rss_url": "https://www.finance-monthly.com/feed/"
|
||||
},
|
||||
"European Financial Review": {
|
||||
"source_website": "europeanfinancialreview.com",
|
||||
"rss_url": "https://www.europeanfinancialreview.com/feed"
|
||||
},
|
||||
"Money Morning": {
|
||||
"source_website": "moneymorning.com",
|
||||
"rss_url": "https://moneymorning.com/feed"
|
||||
},
|
||||
"Dealbreaker": {
|
||||
"source_website": "dealbreaker.com",
|
||||
"rss_url": "https://dealbreaker.com/.rss/full"
|
||||
},
|
||||
"World Finance": {
|
||||
"source_website": "worldfinance.com",
|
||||
"rss_url": "https://www.worldfinance.com/feed"
|
||||
},
|
||||
"Fox Business – Headlines": {
|
||||
"source_website": "foxbusiness.com",
|
||||
"rss_url": "https://moxie.foxbusiness.com/google-publisher/latest.xml"
|
||||
},
|
||||
"FinanceAsia": {
|
||||
"source_website": "financeasia.com",
|
||||
"rss_url": "https://www.financeasia.com/rss/latest"
|
||||
},
|
||||
"CNBC – Business": {
|
||||
"source_website": "cnbc.com",
|
||||
"rss_url": "https://www.cnbc.com/id/100003114/device/rss/rss.html"
|
||||
},
|
||||
"CNN Money": {
|
||||
"source_website": "cnn.com",
|
||||
"rss_url": "http://rss.cnn.com/rss/money_topstories.rss"
|
||||
},
|
||||
"Markets Insider": {
|
||||
"source_website": "markets.businessinsider.com",
|
||||
"rss_url": "https://markets.businessinsider.com/rss/news"
|
||||
},
|
||||
"The Economist – Business & Finance": {
|
||||
"source_website": "economist.com",
|
||||
"rss_url": "https://www.economist.com/business/rss.xml"
|
||||
},
|
||||
"Barchart News": {
|
||||
"source_website": "barchart.com",
|
||||
"rss_url": "http://feeds.feedburner.com/BarchartNews"
|
||||
},
|
||||
"The Guardian – Business": {
|
||||
"source_website": "theguardian.com",
|
||||
"rss_url": "http://feeds.theguardian.com/theguardian/uk/business/rss"
|
||||
},
|
||||
"Economy Watch": {
|
||||
"source_website": "economywatch.com",
|
||||
"rss_url": "https://www.economywatch.com/feed"
|
||||
},
|
||||
"CFI.co": {
|
||||
"source_website": "cfi.co",
|
||||
"rss_url": "https://cfi.co/feed"
|
||||
},
|
||||
"BBC News – Business": {
|
||||
"source_website": "bbc.co.uk",
|
||||
"rss_url": "http://feeds.bbci.co.uk/news/business/rss.xml"
|
||||
},
|
||||
"Investor’s Business Daily": {
|
||||
"source_website": "investors.com",
|
||||
"rss_url": "https://www.investors.com/feed/"
|
||||
},
|
||||
"Forbes – Real-Time": {
|
||||
"source_website": "forbes.com",
|
||||
"rss_url": "https://www.forbes.com/real-time/feed2/"
|
||||
},
|
||||
"The Financial Express": {
|
||||
"source_website": "financialexpress.com",
|
||||
"rss_url": "https://www.financialexpress.com/feed/"
|
||||
},
|
||||
"MarketWatch – Top Stories": {
|
||||
"source_website": "marketwatch.com",
|
||||
"rss_url": "http://feeds.marketwatch.com/marketwatch/topstories/"
|
||||
},
|
||||
"Wall Street Journal – U.S. Business": {
|
||||
"source_website": "wsj.com",
|
||||
"rss_url": "https://feeds.a.dj.com/rss/WSJcomUSBusiness.xml"
|
||||
},
|
||||
"Sky News – Business": {
|
||||
"source_website": "news.sky.com",
|
||||
"rss_url": "http://news.sky.com/feeds/rss/business.xml"
|
||||
},
|
||||
"Bloomberg – Surveillance Podcast": {
|
||||
"source_website": "bloomberg.com",
|
||||
"rss_url": "https://www.bloomberg.com/feed/podcast/bloomberg-surveillance.xml"
|
||||
},
|
||||
"Barron’s – Markets": {
|
||||
"source_website": "barrons.com",
|
||||
"rss_url": "https://www.barrons.com/xml/rss/markets.xml"
|
||||
},
|
||||
"Yahoo Finance": {
|
||||
"source_website": "finance.yahoo.com",
|
||||
"rss_url": "https://www.yahoo.com/news/rss/finance"
|
||||
},
|
||||
"Investing.com – News": {
|
||||
"source_website": "investing.com",
|
||||
"rss_url": "https://www.investing.com/rss/news.rss"
|
||||
},
|
||||
"Investopedia – Headlines": {
|
||||
"source_website": "investopedia.com",
|
||||
"rss_url": "https://www.investopedia.com/feedbuilder/feed/getfeed/?feedName=rss_headline"
|
||||
},
|
||||
"NerdWallet – Finance": {
|
||||
"source_website": "nerdwallet.com",
|
||||
"rss_url": "https://www.nerdwallet.com/news/finance/feed"
|
||||
},
|
||||
"Newsmax Finance": {
|
||||
"source_website": "newsmax.com",
|
||||
"rss_url": "https://www.newsmax.com/rss/finance"
|
||||
},
|
||||
"Bankrate – News": {
|
||||
"source_website": "bankrate.com",
|
||||
"rss_url": "https://www.bankrate.com/rss/"
|
||||
},
|
||||
"Morningstar – Articles": {
|
||||
"source_website": "morningstar.com",
|
||||
"rss_url": "https://www.morningstar.com/articles.rss"
|
||||
},
|
||||
"Kiplinger": {
|
||||
"source_website": "kiplinger.com",
|
||||
"rss_url": "https://www.kiplinger.com/kiplinger.rss"
|
||||
},
|
||||
"International Business Times": {
|
||||
"source_website": "ibtimes.com",
|
||||
"rss_url": "https://www.ibtimes.com/rss"
|
||||
},
|
||||
"Policygenius – News": {
|
||||
"source_website": "policygenius.com",
|
||||
"rss_url": "https://www.policygenius.com/news/feed/"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,8 +0,0 @@
|
||||
{
|
||||
"rss_feeds": {
|
||||
"Associated Press – Business": {
|
||||
"source_website": "apnews.com",
|
||||
"rss_url": "https://news.google.com/rss/search?q=site:apnews.com&hl=en-US&gl=US&ceid=US:en"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,14 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ------------------------------------------------------------------
|
||||
# run_scraper.sh
|
||||
#
|
||||
# Activates the venv *implicitly* by calling the venv’s Python binary.
|
||||
# ------------------------------------------------------------------
|
||||
# Absolute path to the venv – change only if you move the venv.
|
||||
VENV_DIR="/home/user/StockDocs/scraper/venv"
|
||||
|
||||
# Absolute path to the script you want to run.
|
||||
SCRIPT="/home/user/StockDocs/scraper/cron_scraper.py"
|
||||
|
||||
# Invoke the venv’s Python directly.
|
||||
"${VENV_DIR}/bin/python" "${SCRIPT}" # output is redirected by cron
|
||||
@ -1,568 +0,0 @@
|
||||
import newspaper
|
||||
import json
|
||||
import feedparser
|
||||
import time
|
||||
import os
|
||||
import logging
|
||||
import random
|
||||
from datetime import datetime
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.firefox.options import Options as FirefoxOptions
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed, TimeoutError
|
||||
import nltk
|
||||
from nltk.downloader import Downloader
|
||||
|
||||
# Setup logging with timestamps
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
FEED_FILE = os.getenv("FEED_FILE", "./rss_short_feed.json")
|
||||
MAX_FEED_WORKERS = int(os.getenv("MAX_FEED_WORKERS", "10"))
|
||||
MAX_ARTICLE_WORKERS = int(os.getenv("MAX_ARTICLE_WORKERS", "10"))
|
||||
BATCH_SIZE = int(os.getenv("BATCH_SIZE", "50")) # Batch processing size
|
||||
|
||||
# Rotating User-Agents to bypass bot detection (Reuters, etc.)
|
||||
USER_AGENTS = [
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:134.0) Gecko/20100101 Firefox/134.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
]
|
||||
|
||||
def get_random_ua():
|
||||
return random.choice(USER_AGENTS)
|
||||
|
||||
# Ensure necessary NLTK resources are downloaded
|
||||
d = Downloader()
|
||||
if not d.is_installed("punkt_tab"):
|
||||
nltk.download("punkt_tab")
|
||||
|
||||
articles = []
|
||||
|
||||
|
||||
# Enhanced cache system
|
||||
def load_processed_cache():
|
||||
"""Load the processed articles cache with enhanced tracking"""
|
||||
cache_path = "articles/processed_articles_cache.json"
|
||||
try:
|
||||
if os.path.exists(cache_path):
|
||||
with open(cache_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
else:
|
||||
return {}
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading cache: {e}")
|
||||
return {}
|
||||
|
||||
def save_processed_cache(cache_data):
|
||||
"""Save the processed articles cache with enhanced tracking"""
|
||||
cache_path = "articles/processed_articles_cache.json"
|
||||
try:
|
||||
with open(cache_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, indent=2, ensure_ascii=False)
|
||||
logger.info(f"Cache saved with {len(cache_data)} entries")
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving cache: {e}")
|
||||
|
||||
def is_article_processed(article_path, cache_data):
|
||||
"""Check if an article has been processed"""
|
||||
return article_path in cache_data
|
||||
|
||||
def mark_article_processed(article_path, status="completed", embedding_status="pending"):
|
||||
"""Mark an article as processed with detailed status tracking"""
|
||||
cache_data = load_processed_cache()
|
||||
cache_data[article_path] = {
|
||||
"processed_date": datetime.now().isoformat(),
|
||||
"status": status,
|
||||
"embedding_status": embedding_status,
|
||||
"last_updated": datetime.now().isoformat()
|
||||
}
|
||||
save_processed_cache(cache_data)
|
||||
|
||||
def get_processing_progress():
|
||||
"""Get overall processing progress"""
|
||||
cache_data = load_processed_cache()
|
||||
total_articles = len(cache_data)
|
||||
completed_articles = sum(1 for data in cache_data.values() if data.get('status') == 'completed')
|
||||
embedded_articles = sum(1 for data in cache_data.values() if data.get('embedding_status') == 'completed')
|
||||
|
||||
return {
|
||||
"total_articles": total_articles,
|
||||
"completed_articles": completed_articles,
|
||||
"embedded_articles": embedded_articles,
|
||||
"completion_rate": (completed_articles / total_articles * 100) if total_articles > 0 else 0
|
||||
}
|
||||
|
||||
|
||||
def load_rss_feed_sources(feed_file=FEED_FILE):
|
||||
"""
|
||||
Loads the RSS feed sources from a JSON file.
|
||||
"""
|
||||
logger.info(f"Loading RSS feed sources from {feed_file}...")
|
||||
|
||||
try:
|
||||
with open(feed_file, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
logger.info(f"Successfully loaded {feed_file}")
|
||||
return data
|
||||
except FileNotFoundError:
|
||||
logger.error(f"{feed_file} not found, returning empty list.")
|
||||
return []
|
||||
except json.JSONDecodeError:
|
||||
logger.error(f"Error decoding {feed_file} , returning empty list.")
|
||||
return []
|
||||
|
||||
|
||||
def mine_all_articles(rss_feed_sources, limit=None):
|
||||
"""
|
||||
Mines all articles from the given RSS feed sources.
|
||||
Returns a list of (site, title, link) tuples.
|
||||
"""
|
||||
all_links = []
|
||||
|
||||
# Check if rss_feed_sources is a valid dict with rss_feeds key
|
||||
if not isinstance(rss_feed_sources, dict):
|
||||
logger.warning(f"rss_feed_sources is not a dict, it's {type(rss_feed_sources)}")
|
||||
return all_links
|
||||
|
||||
if "rss_feeds" not in rss_feed_sources:
|
||||
logger.warning("rss_feeds key not found in rss_feed_sources")
|
||||
return all_links
|
||||
|
||||
sources = rss_feed_sources["rss_feeds"]
|
||||
|
||||
# Parse RSS feeds in parallel for better performance
|
||||
def parse_single_feed(site, data):
|
||||
"""Parse a single RSS feed with timeout and error handling"""
|
||||
try:
|
||||
logger.info(f"Parsing RSS feed: {data['rss_url']}")
|
||||
# Add more aggressive timeout settings with fallback
|
||||
# Use a wrapper to ensure we don't hang indefinitely
|
||||
import signal
|
||||
|
||||
def timeout_handler(signum, frame):
|
||||
raise TimeoutError(f"Timeout parsing feed: {site}")
|
||||
|
||||
# Set up signal-based timeout (this is a fallback for truly hanging requests)
|
||||
old_handler = signal.signal(signal.SIGALRM, timeout_handler)
|
||||
signal.alarm(10) # 10 second alarm
|
||||
|
||||
feed = feedparser.parse(data["rss_url"], timeout=8) # 8 second timeout
|
||||
signal.alarm(0) # Cancel the alarm
|
||||
signal.signal(signal.SIGALRM, old_handler)
|
||||
|
||||
feed_entries = feed.entries[:limit] if limit else feed.entries
|
||||
|
||||
entries = []
|
||||
for entry in feed_entries:
|
||||
if "link" in entry and "title" in entry:
|
||||
entries.append((site, entry.title, entry.link))
|
||||
return entries
|
||||
except TimeoutError as e:
|
||||
logger.error(f"Timeout parsing RSS feed: {site} Error: {str(e)}")
|
||||
return []
|
||||
except Exception as e:
|
||||
error_str = str(e).lower()
|
||||
# Handle specific network connection issues
|
||||
if "remote end closed connection" in error_str or "connection closed" in error_str:
|
||||
logger.warning(f"Network connection closed by remote end for feed: {site} - {str(e)}")
|
||||
logger.info(f"Skipping problematic feed: {site}")
|
||||
return []
|
||||
elif "timeout" in error_str:
|
||||
logger.error(f"Timeout parsing RSS feed: {site} Error: {str(e)}")
|
||||
return []
|
||||
else:
|
||||
logger.error(f"Error parsing RSS feed: {site} Error: {str(e)}")
|
||||
return []
|
||||
|
||||
# Use ThreadPoolExecutor for parallel RSS feed parsing
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
max_workers = min(10, len(sources)) # Limit concurrent workers
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all feed parsing tasks
|
||||
future_to_site = {
|
||||
executor.submit(parse_single_feed, site, data): site
|
||||
for site, data in sources.items()
|
||||
}
|
||||
|
||||
# Collect results as they complete
|
||||
for future in as_completed(future_to_site, timeout=30): # 30 second overall timeout
|
||||
try:
|
||||
entries = future.result()
|
||||
all_links.extend(entries)
|
||||
except Exception as e:
|
||||
site = future_to_site[future]
|
||||
logger.error(f"Error processing feed for {site}: {str(e)}")
|
||||
|
||||
return all_links
|
||||
|
||||
|
||||
def generate_filename_from_url(url):
|
||||
"""
|
||||
Generates a filename from the given URL by replacing slashes with underscores.
|
||||
"""
|
||||
# Use only the last part of the URL or replace slashes
|
||||
filename = url.replace("https://", "").replace("http://", "").replace("/", "_")
|
||||
# Sanitize filename to remove/replace invalid characters
|
||||
import re
|
||||
filename = re.sub(r'[\\/*?:"<>|]', "_", filename)
|
||||
return filename
|
||||
|
||||
|
||||
def generate_safe_filename(name):
|
||||
# Remove/replace characters not allowed in filenames
|
||||
import re
|
||||
|
||||
safe = re.sub(r'[\\/*?:"<>|]', "_", name)
|
||||
return safe
|
||||
|
||||
|
||||
def save_article_to_file(article, filename, source="Unfiltered"):
|
||||
"""
|
||||
Saves the given article text to a file with the specified filename.
|
||||
"""
|
||||
# articles dir should already be there
|
||||
# os.makedirs("articles", exist_ok=True)
|
||||
|
||||
outputDir = "articles/" + source
|
||||
os.makedirs(outputDir, exist_ok=True) if source else None
|
||||
|
||||
# Sanitize filename: use only the last part of the URL or replace slashes
|
||||
safe_filename = generate_filename_from_url(filename)
|
||||
file_path = os.path.join(outputDir, safe_filename)
|
||||
# Save the source as the first line in the file for later retrieval
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
f.write(f"SOURCE:{source}\n")
|
||||
f.write(article)
|
||||
|
||||
# Only log when a new file is actually created (not cached)
|
||||
logger.info(f"New article saved: {safe_filename} from {source}")
|
||||
|
||||
|
||||
def get_article_with_selenium(url):
|
||||
"""
|
||||
Gets article text using Selenium Firefox driver with proper error handling,
|
||||
cleanup, and bot-detection evasion.
|
||||
"""
|
||||
driver = None
|
||||
try:
|
||||
# Configure Firefox options with bot-detection evasion
|
||||
options = FirefoxOptions()
|
||||
options.add_argument("--headless")
|
||||
options.set_preference("dom.ipc.processCount", 1)
|
||||
options.set_preference("general.useragent.override", get_random_ua())
|
||||
options.set_preference("permissions.default.image", 2) # Block images for speed
|
||||
options.set_preference("dom.webnotifications.enabled", False)
|
||||
|
||||
# Try to initialize driver with explicit path to Firefox
|
||||
try:
|
||||
driver = webdriver.Firefox(options=options)
|
||||
except Exception as e:
|
||||
# If that fails, try with explicit Firefox path
|
||||
if "binary is not a firefox executable" in str(e).lower():
|
||||
logger.info("Attempting to use Firefox at /usr/bin/firefox")
|
||||
options.binary_location = "/usr/bin/firefox"
|
||||
driver = webdriver.Firefox(options=options)
|
||||
else:
|
||||
raise e
|
||||
|
||||
driver.set_page_load_timeout(30) # 30 seconds timeout
|
||||
|
||||
# Navigate to URL
|
||||
driver.get(url)
|
||||
|
||||
# Wait for page to load (explicit wait instead of sleep)
|
||||
try:
|
||||
WebDriverWait(driver, 15).until(
|
||||
EC.presence_of_element_located((By.TAG_NAME, "body"))
|
||||
)
|
||||
except Exception:
|
||||
pass # Continue even if wait times out
|
||||
|
||||
time.sleep(random.uniform(1, 3)) # Random wait to mimic human behavior
|
||||
|
||||
html = driver.page_source
|
||||
|
||||
# Parse with Newspaper4k
|
||||
article = newspaper.article(url, input_html=html, language="en")
|
||||
article.nlp()
|
||||
logger.info(f"Successfully extracted article with Selenium from {url}")
|
||||
return article.text
|
||||
|
||||
except Exception as e:
|
||||
# Check if this is a Firefox binary not found error
|
||||
error_str = str(e).lower()
|
||||
if "binary is not a firefox executable" in error_str or "firefox" in error_str:
|
||||
logger.error(f"Firefox not found or not properly configured for {url}: {str(e)}")
|
||||
logger.error("Firefox is installed at /usr/bin/firefox but may not be accessible. Check PATH or permissions.")
|
||||
else:
|
||||
logger.error(f"Selenium failed for {url}: {str(e)}")
|
||||
return ""
|
||||
finally:
|
||||
# Always quit the driver
|
||||
if driver:
|
||||
try:
|
||||
driver.quit()
|
||||
except Exception:
|
||||
pass # Ignore errors in cleanup
|
||||
|
||||
|
||||
def get_article_with_playwright(url):
|
||||
"""
|
||||
Gets article text using Playwright with proper bot-detection evasion.
|
||||
"""
|
||||
try:
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
with sync_playwright() as p:
|
||||
# Use Chromium with full browser context for UA spoofing
|
||||
browser = p.chromium.launch(headless=True, timeout=30000)
|
||||
context = browser.new_context(
|
||||
user_agent=get_random_ua(),
|
||||
viewport={"width": 1920, "height": 1080},
|
||||
locale="en-US",
|
||||
timezone_id="America/New_York",
|
||||
)
|
||||
page = context.new_page()
|
||||
|
||||
# Additional headers for legitimacy
|
||||
page.set_extra_http_headers({
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"Connection": "keep-alive",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
})
|
||||
|
||||
page.goto(url, wait_until="domcontentloaded", timeout=30000)
|
||||
|
||||
# Wait for content to load
|
||||
time.sleep(random.uniform(2, 4))
|
||||
|
||||
html = page.content()
|
||||
context.close()
|
||||
browser.close()
|
||||
|
||||
# Parse with Newspaper4k
|
||||
article = newspaper.article(url, input_html=html, language="en")
|
||||
article.nlp()
|
||||
logger.info(f"Successfully extracted article with Playwright from {url}")
|
||||
return article.text
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Playwright failed for {url}: {str(e)}")
|
||||
return ""
|
||||
|
||||
|
||||
def pull_article(link, source, title=None, save_to_file=True):
|
||||
"""
|
||||
Pulls an article from a given link with fallback mechanisms.
|
||||
"""
|
||||
filename = title if title else link
|
||||
safe_filename = generate_filename_from_url(filename)
|
||||
|
||||
# Check if already cached
|
||||
if os.path.exists(os.path.join("articles", source, safe_filename)):
|
||||
logger.info(f"Article already cached: {filename}")
|
||||
with open(
|
||||
os.path.join("articles", source, safe_filename), "r", encoding="utf-8"
|
||||
) as f:
|
||||
return f.read()
|
||||
|
||||
# Random delay before fetching to avoid rate-limiting / bot detection
|
||||
time.sleep(random.uniform(0.5, 2))
|
||||
|
||||
text = ""
|
||||
|
||||
try:
|
||||
# Try newspaper4k first with proper User-Agent to bypass bot detection
|
||||
ua = get_random_ua()
|
||||
article = newspaper.article(link, browser_user_agent=ua)
|
||||
article.download()
|
||||
article.parse()
|
||||
text = article.text
|
||||
|
||||
if not text or len(text) < 200:
|
||||
raise ValueError(
|
||||
"\tArticle text too short, falling back to Playwright/Selenium."
|
||||
)
|
||||
logger.info(f"Successfully pulled article with newspaper4k from {link}")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"newspaper4k extraction failed for {link}: {e}, falling back to Playwright."
|
||||
)
|
||||
|
||||
try:
|
||||
text = get_article_with_playwright(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Playwright")
|
||||
|
||||
if not text or len(text) < 200:
|
||||
logger.warning("Playwright article too short, falling back to Selenium.")
|
||||
# Fallback to Selenium with better error handling
|
||||
text = get_article_with_selenium(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Selenium")
|
||||
except Exception as e:
|
||||
logger.error(f"Playwright failed for {link}: {e}")
|
||||
|
||||
# Fallback to Selenium
|
||||
try:
|
||||
text = get_article_with_selenium(link)
|
||||
logger.info(f"Successfully pulled article from {link} with Selenium")
|
||||
except Exception as e:
|
||||
logger.error(f"Selenium failed for {link}: {e}")
|
||||
return ""
|
||||
|
||||
if save_to_file:
|
||||
save_article_to_file(text, filename, source)
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def is_article_downloaded(source, title, link):
|
||||
"""
|
||||
Check if an article is already downloaded by checking if the file exists.
|
||||
"""
|
||||
# Generate the same filename that would be used for saving
|
||||
filename = title if title else link
|
||||
safe_filename = generate_filename_from_url(filename)
|
||||
|
||||
# Check if file exists in the articles directory
|
||||
file_path = os.path.join("articles", source, safe_filename)
|
||||
return os.path.exists(file_path)
|
||||
|
||||
|
||||
def safe_pull_articles(article_list):
|
||||
"""
|
||||
Safely pull articles with improved error handling and increased parallelism.
|
||||
"""
|
||||
if not article_list:
|
||||
return [], []
|
||||
|
||||
# Filter out articles that are already downloaded
|
||||
filtered_article_list = []
|
||||
total_articles = len(article_list)
|
||||
|
||||
for source, title, link in article_list:
|
||||
if not is_article_downloaded(source, title, link):
|
||||
filtered_article_list.append((source, title, link))
|
||||
else:
|
||||
logger.info(f"Skipping already downloaded article: {title[:50]}... from {source}")
|
||||
|
||||
logger.info(f"Filtered out {total_articles - len(filtered_article_list)} articles that were already downloaded")
|
||||
logger.info(f"Processing {len(filtered_article_list)} remaining articles")
|
||||
|
||||
if not filtered_article_list:
|
||||
logger.info("No new articles to process")
|
||||
return [], []
|
||||
|
||||
results = []
|
||||
errors = []
|
||||
|
||||
# Use ThreadPoolExecutor for parallel article pulling with higher concurrency
|
||||
# Use configurable worker setting
|
||||
max_workers = min(MAX_ARTICLE_WORKERS, len(filtered_article_list)) # Cap at configured workers, but don't exceed article count
|
||||
batch_size = max(1, min(20, len(filtered_article_list) // 4)) # Dynamic batch size
|
||||
|
||||
logger.info(f"Starting parallel article pulling with {max_workers} workers and batch size {batch_size}")
|
||||
|
||||
# Process all articles in parallel with proper error handling
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all tasks at once for maximum parallelism
|
||||
futures = [
|
||||
executor.submit(pull_article, link, source, title)
|
||||
for source, title, link in filtered_article_list
|
||||
]
|
||||
|
||||
# Collect results as they complete
|
||||
for i, future in enumerate(as_completed(futures, timeout=300)): # 5 minute timeout total
|
||||
try:
|
||||
result = future.result(timeout=120) # 2 minute timeout per article
|
||||
if result: # Only count non-empty results
|
||||
results.append(result)
|
||||
# Log progress every 100 articles with proper batch information
|
||||
if (i + 1) % 100 == 0:
|
||||
logger.info(f"Processed {i + 1} articles out of {len(filtered_article_list)}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing article: {e}")
|
||||
errors.append(e)
|
||||
|
||||
return results, errors
|
||||
|
||||
|
||||
def main():
|
||||
"""
|
||||
Main scraping loop.
|
||||
"""
|
||||
while True:
|
||||
logger.info("=========================================")
|
||||
logger.info("Starting new scraping iteration...")
|
||||
|
||||
try:
|
||||
# Pull the RSS feed sources from the JSON file
|
||||
rss_feed_sources = load_rss_feed_sources()
|
||||
|
||||
# Mine all articles from the RSS feed sources
|
||||
rss_feed_links = mine_all_articles(rss_feed_sources)
|
||||
|
||||
# Randomize the order of the links to help with load balancing
|
||||
import random
|
||||
|
||||
random.shuffle(rss_feed_links)
|
||||
|
||||
logger.info(f"Found {len(rss_feed_links)} articles to process")
|
||||
|
||||
if not rss_feed_links:
|
||||
logger.info("No articles found, sleeping for 15 minutes")
|
||||
time.sleep(15 * 60)
|
||||
continue
|
||||
|
||||
# Process articles with better error handling and resource management
|
||||
results, errors = safe_pull_articles(rss_feed_links)
|
||||
|
||||
logger.info(f"Attempted to Pull {len(results)} articles in parallel.")
|
||||
logger.info(
|
||||
f"Encountered {len(errors)} errors during article pulling. "
|
||||
+ "Outputting errors to a local file."
|
||||
)
|
||||
|
||||
# Output errors to a local file
|
||||
if errors:
|
||||
with open("errors.txt", "w", encoding="utf-8") as f:
|
||||
for error in errors:
|
||||
f.write(str(error) + "\n")
|
||||
logger.info("Errors logged to errors.txt")
|
||||
|
||||
# Print all results to a log file
|
||||
with open("results.txt", "w", encoding="utf-8") as f:
|
||||
for result in results:
|
||||
f.write(result + "\n")
|
||||
logger.info("All articles pulled successfully.")
|
||||
|
||||
# Log processing progress
|
||||
progress = get_processing_progress()
|
||||
logger.info(f"Processing progress - Total: {progress['total_articles']}, "
|
||||
f"Completed: {progress['completed_articles']}, "
|
||||
f"Embedded: {progress['embedded_articles']}, "
|
||||
f"Completion rate: {progress['completion_rate']:.1f}%")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Major error in main loop: {e}")
|
||||
# Continue to next iteration even if there's a major error
|
||||
|
||||
# Sleep for a while before the next iteration
|
||||
logger.info("Sleeping for 15 minutes before the next iteration...")
|
||||
time.sleep(15 * 60) # Sleep for 15 minutes
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,91 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ------------------------------------------------------------------
|
||||
# setup_scraper.sh
|
||||
#
|
||||
# Installs a virtual‑env, a tiny wrapper, optional dependencies
|
||||
# and an hourly cron job – *always using the venv’s python/pip*.
|
||||
# ------------------------------------------------------------------
|
||||
set -euo pipefail
|
||||
|
||||
# ---------- Configuration ----------
|
||||
SCRAPER_DIR="/home/user/StockDocs/scraper"
|
||||
VENV_DIR="${SCRAPER_DIR}/venv"
|
||||
SCRIPT_PATH="${SCRAPER_DIR}/cron_scraper.py"
|
||||
WRAPPER_PATH="${SCRAPER_DIR}/run_scraper.sh"
|
||||
LOG_PATH="${SCRAPER_DIR}/cron.log"
|
||||
REQUIREMENTS="${SCRAPER_DIR}/requirements.txt"
|
||||
|
||||
# Cron line – this will call the wrapper, which in turn calls the
|
||||
# venv’s Python interpreter.
|
||||
CRON_LINE="0 * * * * ${WRAPPER_PATH} >> ${LOG_PATH} 2>&1"
|
||||
|
||||
# ---------- Helper functions ----------
|
||||
log() { printf '[setup_scraper] %s\n' "$*"; }
|
||||
error_exit() { printf '[setup_scraper] ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# ---------- 1️⃣ Create the venv if it’s missing ----------
|
||||
if [[ ! -d "$VENV_DIR" ]]; then
|
||||
log "Creating virtual‑environment at ${VENV_DIR}"
|
||||
python3 -m venv "$VENV_DIR" || error_exit "Failed to create venv"
|
||||
else
|
||||
log "Virtual‑environment already exists at ${VENV_DIR}"
|
||||
fi
|
||||
|
||||
# ---------- 2️⃣ Activate the venv *for the rest of this script* ----------
|
||||
# This changes PATH for the current shell only – it does NOT touch the
|
||||
# system Python. The next line will confirm that we’re really in the venv.
|
||||
# shellcheck source=/dev/null
|
||||
source "${VENV_DIR}/bin/activate"
|
||||
|
||||
# Quick sanity‑check: make sure we’re using the venv’s python and pip.
|
||||
log "Current python: $(python -c 'import sys;print(sys.executable)')"
|
||||
log "pip version: $(pip --version)"
|
||||
|
||||
# ---------- 3️⃣ Install dependencies (optional) ----------
|
||||
if [[ -f "$REQUIREMENTS" ]]; then
|
||||
log "Installing Python packages from ${REQUIREMENTS}"
|
||||
# Use the *venv’s* pip explicitly – this guarantees no system installs.
|
||||
"${VENV_DIR}/bin/pip" install --upgrade pip
|
||||
"${VENV_DIR}/bin/pip" install -r "$REQUIREMENTS" || error_exit "pip install failed"
|
||||
else
|
||||
log "No requirements.txt found – skipping dependency install."
|
||||
fi
|
||||
|
||||
# ---------- 4️⃣ Create the wrapper script ----------
|
||||
# The wrapper *does NOT* source the venv any more – it calls the venv’s
|
||||
# Python binary directly. This eliminates the subtle “activate” pitfall.
|
||||
if [[ ! -f "$WRAPPER_PATH" ]]; then
|
||||
log "Creating wrapper script at ${WRAPPER_PATH}"
|
||||
cat > "$WRAPPER_PATH" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
# ------------------------------------------------------------------
|
||||
# run_scraper.sh
|
||||
#
|
||||
# Activates the venv *implicitly* by calling the venv’s Python binary.
|
||||
# ------------------------------------------------------------------
|
||||
# Absolute path to the venv – change only if you move the venv.
|
||||
VENV_DIR="/home/user/StockDocs/scraper/venv"
|
||||
|
||||
# Absolute path to the script you want to run.
|
||||
SCRIPT="/home/user/StockDocs/scraper/cron_scraper.py"
|
||||
|
||||
# Invoke the venv’s Python directly.
|
||||
"${VENV_DIR}/bin/python" "${SCRIPT}" # output is redirected by cron
|
||||
EOF
|
||||
chmod +x "$WRAPPER_PATH"
|
||||
else
|
||||
log "Wrapper script already exists at ${WRAPPER_PATH}"
|
||||
fi
|
||||
|
||||
# ---------- 5️⃣ Install the cron job ----------
|
||||
CURRENT_CRON=$(crontab -l 2>/dev/null || true)
|
||||
|
||||
if echo "$CURRENT_CRON" | grep -Fqx "$CRON_LINE"; then
|
||||
log "Crontab entry already present – nothing to do."
|
||||
else
|
||||
log "Adding new cron entry."
|
||||
# Append the new line and reinstall the crontab.
|
||||
(printf '%s\n' "$CURRENT_CRON" ; printf '%s\n' "$CRON_LINE") | crontab -
|
||||
fi
|
||||
|
||||
log "✅ Setup complete! ${SCRIPT_PATH} will run every hour via the venv."
|
||||
@ -1,61 +0,0 @@
|
||||
"""Tests for the scraper article-processing cache."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from scraper.scraper import ( # noqa: E402
|
||||
get_processing_progress,
|
||||
load_processed_cache,
|
||||
mark_article_processed,
|
||||
save_processed_cache,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def cache_dir(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
(tmp_path / "articles").mkdir()
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_load_missing_cache_returns_empty(cache_dir):
|
||||
assert load_processed_cache() == {}
|
||||
|
||||
|
||||
def test_save_and_load_roundtrip(cache_dir):
|
||||
save_processed_cache({"a.html": {"status": "completed"}})
|
||||
cache = load_processed_cache()
|
||||
assert cache["a.html"]["status"] == "completed"
|
||||
|
||||
|
||||
def test_mark_article_processed(cache_dir):
|
||||
mark_article_processed("articles/x/article.html")
|
||||
cache = load_processed_cache()
|
||||
entry = cache["articles/x/article.html"]
|
||||
assert entry["status"] == "completed"
|
||||
assert entry["embedding_status"] == "pending"
|
||||
assert "processed_date" in entry
|
||||
assert "last_updated" in entry
|
||||
|
||||
|
||||
def test_processing_progress_empty(cache_dir):
|
||||
progress = get_processing_progress()
|
||||
assert progress["total_articles"] == 0
|
||||
assert progress["completed_articles"] == 0
|
||||
assert progress["embedded_articles"] == 0
|
||||
assert progress["completion_rate"] == 0
|
||||
|
||||
|
||||
def test_processing_progress_mixed(cache_dir):
|
||||
mark_article_processed("a.html")
|
||||
mark_article_processed("b.html", status="failed")
|
||||
mark_article_processed("c.html", embedding_status="completed")
|
||||
progress = get_processing_progress()
|
||||
assert progress["total_articles"] == 3
|
||||
assert progress["completed_articles"] == 2
|
||||
assert progress["embedded_articles"] == 1
|
||||
assert progress["completion_rate"] == pytest.approx(2 / 3 * 100)
|
||||
Loading…
x
Reference in New Issue
Block a user