gutting old ai_processor code that's not usful
This commit is contained in:
parent
9ab3e459e5
commit
cb6c42fa65
@ -1,118 +0,0 @@
|
|||||||
# AI Processor
|
|
||||||
|
|
||||||
An AI-powered analytics engine designed to process financial news articles and extract meaningful insights, sentiment analysis, and market indicators from collected content.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The AI Processor is the intelligent component of the system that analyzes the financial news articles collected by the scraper. It uses natural language processing techniques and machine learning models to extract key insights, determine sentiment, identify market trends, and generate actionable analytics for traders and investors.
|
|
||||||
|
|
||||||
## Project Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
ai_processor/
|
|
||||||
├── app.py # Main AI processing application
|
|
||||||
├── config.py # Configuration settings
|
|
||||||
├── requirements.txt # Python dependencies
|
|
||||||
├── Dockerfile # Docker configuration
|
|
||||||
├── README.md # This file
|
|
||||||
├── models/ # Machine learning models and NLP components
|
|
||||||
│ ├── __init__.py
|
|
||||||
│ ├── sentiment_analyzer.py # Sentiment analysis module
|
|
||||||
│ ├── topic_classifier.py # News categorization module
|
|
||||||
│ └── entity_extractor.py # Named entity recognition
|
|
||||||
├── processors/ # Article processing pipelines
|
|
||||||
│ ├── __init__.py
|
|
||||||
│ ├── text_processor.py # Text cleaning and preprocessing
|
|
||||||
│ └── analysis_pipeline.py # Full analysis pipeline
|
|
||||||
└── data/ # Processed data storage
|
|
||||||
├── insights/
|
|
||||||
└── reports/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- **Sentiment Analysis**: Determine positive, negative, or neutral sentiment of news articles
|
|
||||||
- **Topic Classification**: Categorize articles by financial topics (economics, politics, technology, etc.)
|
|
||||||
- **Entity Extraction**: Identify key entities mentioned in articles (stocks, companies, people, organizations)
|
|
||||||
- **Market Indicator Detection**: Extract quantitative indicators that may affect stock prices
|
|
||||||
- **Insight Generation**: Automated generation of actionable intelligence from news content
|
|
||||||
- **Batch Processing**: Process large volumes of articles efficiently
|
|
||||||
|
|
||||||
## Endpoints
|
|
||||||
|
|
||||||
### Article Analysis
|
|
||||||
- POST `/api/analyze/article` - Analyze a single article for insights
|
|
||||||
- POST `/api/analyze/batch` - Process multiple articles in batch mode
|
|
||||||
- GET `/api/analyze/status/{task_id}` - Check processing status
|
|
||||||
|
|
||||||
### Data Access
|
|
||||||
- GET `/api/insights/latest` - Get latest analysis insights
|
|
||||||
- GET `/api/insights/articles/{article_path}` - Get insights for specific article
|
|
||||||
- GET `/api/reports/generate` - Generate comprehensive market analysis report
|
|
||||||
|
|
||||||
### Model Management
|
|
||||||
- GET `/api/models` - List available AI models
|
|
||||||
- POST `/api/models/update` - Update or retrain models with new data
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
The AI Processor supports configuration through environment variables:
|
|
||||||
|
|
||||||
- `MODEL_PATH` - Path to pre-trained AI models (default: `models/`)
|
|
||||||
- `ARTICLE_DIR` - Directory containing articles to process (default: `../scraper/articles`)
|
|
||||||
- `ENABLE_CACHING` - Enable/disable result caching (default: True)
|
|
||||||
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
|
||||||
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
1. Install dependencies:
|
|
||||||
```bash
|
|
||||||
pip install -r requirements.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Set up environment variables (optional but recommended):
|
|
||||||
```bash
|
|
||||||
export MODEL_PATH="/path/to/models"
|
|
||||||
export ARTICLE_DIR="/path/to/articles"
|
|
||||||
export ENABLE_CACHING=true
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Run the AI processor:
|
|
||||||
```bash
|
|
||||||
python app.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## Usage Examples
|
|
||||||
|
|
||||||
### Analyze a single article:
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://localhost:5001/api/analyze/article" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"path":"/path/to/article.txt","source":"Reuters – Business News"}'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Batch process articles:
|
|
||||||
```bash
|
|
||||||
curl -X POST "http://localhost:5001/api/analyze/batch" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"article_paths":["/path/to/article1.txt","/path/to/article2.txt"],"include_sentiment":true}'
|
|
||||||
```
|
|
||||||
|
|
||||||
### Get latest insights:
|
|
||||||
```bash
|
|
||||||
curl "http://localhost:5001/api/insights/latest?limit=10"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
- Python 3.6+
|
|
||||||
- NLP libraries (spaCy, NLTK, transformers)
|
|
||||||
- Machine learning frameworks (scikit-learn, tensorflow/PyTorch)
|
|
||||||
- Additional dependencies listed in `requirements.txt`
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
This project is licensed under the MIT License.
|
|
||||||
@ -1,182 +0,0 @@
|
|||||||
import os
|
|
||||||
import requests
|
|
||||||
import json
|
|
||||||
import datetime
|
|
||||||
import time
|
|
||||||
|
|
||||||
# Simplified AI processor for fact extraction
|
|
||||||
# This version focuses on the core fact extraction functionality
|
|
||||||
|
|
||||||
# AI Service endpoint
|
|
||||||
AI_SERVER_HOST = os.getenv("AI_SERVER_HOST", "example.com")
|
|
||||||
AI_SERVER_PORT = os.getenv("AI_SERVER_PORT", "4000")
|
|
||||||
AI_SERVER_URL = f"http://{AI_SERVER_HOST}:{AI_SERVER_PORT}"
|
|
||||||
|
|
||||||
# API Key for AI service authentication
|
|
||||||
AI_SERVICE_API_KEY = os.getenv("AI_SERVICE_API_KEY")
|
|
||||||
|
|
||||||
def process_article_content(article_content, filename, source):
|
|
||||||
"""
|
|
||||||
Process article content and extract key facts using the centralized AI service
|
|
||||||
This is the core fact extraction function
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
# Use the gpt-oss model for fact extraction as specified
|
|
||||||
extraction_url = f"{AI_SERVER_URL}/v1/chat/completions"
|
|
||||||
|
|
||||||
# Build headers with authentication if available
|
|
||||||
headers = {
|
|
||||||
"Content-Type": "application/json"
|
|
||||||
}
|
|
||||||
|
|
||||||
if AI_SERVICE_API_KEY:
|
|
||||||
headers["Authorization"] = f"Bearer {AI_SERVICE_API_KEY}"
|
|
||||||
|
|
||||||
# Create a proper prompt for fact extraction
|
|
||||||
prompt = f"""
|
|
||||||
Extract key facts from the following article in structured JSON format.
|
|
||||||
Return only valid JSON without any additional text.
|
|
||||||
|
|
||||||
Article Title: {filename}
|
|
||||||
Article Content: {article_content[:2000]}...
|
|
||||||
|
|
||||||
Extract the following information:
|
|
||||||
1. Main topic/subject
|
|
||||||
2. Key entities (companies, people, locations, organizations)
|
|
||||||
3. Financial impact or implications
|
|
||||||
4. Key dates or time periods mentioned
|
|
||||||
5. Summary of main points
|
|
||||||
|
|
||||||
Format the response as a JSON object with these fields:
|
|
||||||
{{
|
|
||||||
"filename": "{filename}",
|
|
||||||
"source": "{source}",
|
|
||||||
"original_content": "{article_content[:1000]}...",
|
|
||||||
"extracted_facts": {{
|
|
||||||
"summary": "brief summary",
|
|
||||||
"key_entities": ["entity1", "entity2"],
|
|
||||||
"financial_impact": "positive/negative/neutral",
|
|
||||||
"main_topic": "main topic",
|
|
||||||
"key_dates": ["date1", "date2"],
|
|
||||||
"main_points": ["point1", "point2", "point3"]
|
|
||||||
}},
|
|
||||||
"processed_at": "{datetime.datetime.now().isoformat()}"
|
|
||||||
}}
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Call the AI service with gpt-oss model
|
|
||||||
response = requests.post(
|
|
||||||
extraction_url,
|
|
||||||
json={
|
|
||||||
"model": "gpt-oss",
|
|
||||||
"messages": [
|
|
||||||
{"role": "system", "content": "You are a helpful assistant that extracts structured facts from articles."},
|
|
||||||
{"role": "user", "content": prompt}
|
|
||||||
],
|
|
||||||
"temperature": 0.3,
|
|
||||||
"max_tokens": 1000
|
|
||||||
},
|
|
||||||
headers=headers,
|
|
||||||
timeout=60
|
|
||||||
)
|
|
||||||
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
# Parse the response
|
|
||||||
result = response.json()
|
|
||||||
extracted_text = result['choices'][0]['message']['content'].strip()
|
|
||||||
|
|
||||||
# Try to parse the JSON from the response
|
|
||||||
try:
|
|
||||||
facts = json.loads(extracted_text)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
# If JSON parsing fails, create a basic structure
|
|
||||||
facts = {
|
|
||||||
"filename": filename,
|
|
||||||
"source": source,
|
|
||||||
"original_content": article_content[:1000] + "..." if len(article_content) > 1000 else article_content,
|
|
||||||
"extracted_facts": {
|
|
||||||
"summary": article_content[:200] + "..." if len(article_content) > 200 else article_content,
|
|
||||||
"key_entities": ["Sample Company", "Sample Person"],
|
|
||||||
"financial_impact": "neutral",
|
|
||||||
"main_topic": "Business/Financial News",
|
|
||||||
"key_dates": ["2026"],
|
|
||||||
"main_points": ["Sample point 1", "Sample point 2"]
|
|
||||||
},
|
|
||||||
"processed_at": datetime.datetime.now().isoformat()
|
|
||||||
}
|
|
||||||
|
|
||||||
return facts
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error processing article {filename}: {e}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
def main_fact_extraction_loop():
|
|
||||||
"""
|
|
||||||
Main loop for fact extraction - this should be called by the embedding pipeline
|
|
||||||
"""
|
|
||||||
print("Starting fact extraction loop...")
|
|
||||||
|
|
||||||
# Retrieve the current archive of pulled articles
|
|
||||||
# Use the correct path for the scraper articles directory
|
|
||||||
articles_folder = os.path.join("articles")
|
|
||||||
if not os.path.exists(articles_folder):
|
|
||||||
print(f"Articles folder {articles_folder} does not exist. Please check the path.")
|
|
||||||
return
|
|
||||||
|
|
||||||
print("Loading articles from folder " + articles_folder + " ...")
|
|
||||||
|
|
||||||
# Process articles from the scraper directory
|
|
||||||
processed_count = 0
|
|
||||||
failed_count = 0
|
|
||||||
|
|
||||||
# Walk through all subdirectories in articles folder
|
|
||||||
for root, dirs, files in os.walk(articles_folder):
|
|
||||||
for filename in files:
|
|
||||||
# Only process text files (not the cache file)
|
|
||||||
if filename == "processed_articles_cache.json":
|
|
||||||
continue
|
|
||||||
|
|
||||||
file_path = os.path.join(root, filename)
|
|
||||||
# Create output path in the output directory
|
|
||||||
output_path = os.path.join("output", f"{filename}.json")
|
|
||||||
|
|
||||||
# Skip if already processed
|
|
||||||
if os.path.isfile(output_path):
|
|
||||||
print(f"Skipping already processed article: {filename}")
|
|
||||||
continue
|
|
||||||
|
|
||||||
if os.path.isfile(file_path):
|
|
||||||
try:
|
|
||||||
with open(file_path, 'r', encoding='utf-8') as f:
|
|
||||||
first_line = f.readline()
|
|
||||||
if first_line.startswith("SOURCE:"):
|
|
||||||
source = first_line[len("SOURCE:"):].strip()
|
|
||||||
content = f.read()
|
|
||||||
else:
|
|
||||||
source = "Unfiltered"
|
|
||||||
content = first_line + f.read()
|
|
||||||
|
|
||||||
# Process the article
|
|
||||||
facts = process_article_content(content, filename, source)
|
|
||||||
|
|
||||||
if facts:
|
|
||||||
# Save the processed result to a JSON file
|
|
||||||
os.makedirs("output", exist_ok=True)
|
|
||||||
with open(output_path, "w", encoding="utf-8") as f:
|
|
||||||
json.dump(facts, f, ensure_ascii=False, indent=2)
|
|
||||||
processed_count += 1
|
|
||||||
print(f"Processed and saved: {filename}")
|
|
||||||
else:
|
|
||||||
failed_count += 1
|
|
||||||
print(f"Failed to process: {filename}")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Error processing article {filename}: {e}")
|
|
||||||
failed_count += 1
|
|
||||||
|
|
||||||
print(f"Fact extraction complete. Processed: {processed_count}, Failed: {failed_count}")
|
|
||||||
|
|
||||||
# Run once when called directly
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main_fact_extraction_loop()
|
|
||||||
@ -1,16 +0,0 @@
|
|||||||
FROM python:3.11-slim
|
|
||||||
|
|
||||||
WORKDIR /app
|
|
||||||
|
|
||||||
# Copy requirements first for better caching
|
|
||||||
COPY requirements.txt .
|
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
|
||||||
|
|
||||||
# Copy the rest of the application
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# Create output directory
|
|
||||||
RUN mkdir -p output
|
|
||||||
|
|
||||||
# Command to run the processor
|
|
||||||
CMD ["python", "ai_processor.py"]
|
|
||||||
@ -1,3 +0,0 @@
|
|||||||
home = /Library/Developer/CommandLineTools/usr/bin
|
|
||||||
include-system-site-packages = false
|
|
||||||
version = 3.9.6
|
|
||||||
@ -1,3 +0,0 @@
|
|||||||
requests
|
|
||||||
numpy
|
|
||||||
python-dotenv
|
|
||||||
Loading…
x
Reference in New Issue
Block a user