StockDocs/ai_processor

AI Processor

An AI-powered analytics engine designed to process financial news articles and extract meaningful insights, sentiment analysis, and market indicators from collected content.

Overview

The AI Processor is the intelligent component of the system that analyzes the financial news articles collected by the scraper. It uses natural language processing techniques and machine learning models to extract key insights, determine sentiment, identify market trends, and generate actionable analytics for traders and investors.

Project Structure

ai_processor/
├── app.py                 # Main AI processing application
├── config.py              # Configuration settings
├── requirements.txt       # Python dependencies
├── Dockerfile             # Docker configuration  
├── README.md              # This file
├── models/                # Machine learning models and NLP components
│   ├── __init__.py
│   ├── sentiment_analyzer.py  # Sentiment analysis module
│   ├── topic_classifier.py    # News categorization module
│   └── entity_extractor.py    # Named entity recognition
├── processors/            # Article processing pipelines
│   ├── __init__.py
│   ├── text_processor.py      # Text cleaning and preprocessing
│   └── analysis_pipeline.py   # Full analysis pipeline
└── data/                  # Processed data storage  
    ├── insights/
    └── reports/

Features

• Sentiment Analysis: Determine positive, negative, or neutral sentiment of news articles
• Topic Classification: Categorize articles by financial topics (economics, politics, technology, etc.)
• Entity Extraction: Identify key entities mentioned in articles (stocks, companies, people, organizations)
• Market Indicator Detection: Extract quantitative indicators that may affect stock prices
• Insight Generation: Automated generation of actionable intelligence from news content
• Batch Processing: Process large volumes of articles efficiently

Endpoints

Article Analysis

• POST /api/analyze/article - Analyze a single article for insights
• POST /api/analyze/batch - Process multiple articles in batch mode
• GET /api/analyze/status/{task_id} - Check processing status

Data Access

• GET /api/insights/latest - Get latest analysis insights
• GET /api/insights/articles/{article_path} - Get insights for specific article
• GET /api/reports/generate - Generate comprehensive market analysis report

Model Management

• GET /api/models - List available AI models
• POST /api/models/update - Update or retrain models with new data

Configuration

Environment Variables

The AI Processor supports configuration through environment variables:

• MODEL_PATH - Path to pre-trained AI models (default: models/)
• ARTICLE_DIR - Directory containing articles to process (default: ../scraper/articles)
• ENABLE_CACHING - Enable/disable result caching (default: True)
• LOG_LEVEL - Logging level (DEBUG, INFO, WARNING, ERROR)
• MAX_WORKERS - Number of concurrent processing threads (default: 4)

Installation

1. Install dependencies:

pip install -r requirements.txt

2. Set up environment variables (optional but recommended):

export MODEL_PATH="/path/to/models"
export ARTICLE_DIR="/path/to/articles"
export ENABLE_CACHING=true

3. Run the AI processor:

python app.py

Usage Examples

Analyze a single article:

curl -X POST "http://localhost:5001/api/analyze/article" \
  -H "Content-Type: application/json" \
  -d '{"path":"/path/to/article.txt","source":"Reuters – Business News"}'

Batch process articles:

curl -X POST "http://localhost:5001/api/analyze/batch" \
  -H "Content-Type: application/json" \
  -d '{"article_paths":["/path/to/article1.txt","/path/to/article2.txt"],"include_sentiment":true}'

Get latest insights:

curl "http://localhost:5001/api/insights/latest?limit=10"

Requirements

• Python 3.6+
• NLP libraries (spaCy, NLTK, transformers)
• Machine learning frameworks (scikit-learn, tensorflow/PyTorch)
• Additional dependencies listed in requirements.txt

License

This project is licensed under the MIT License.