118 lines
4.2 KiB
Markdown
118 lines
4.2 KiB
Markdown
# AI Processor
|
||
|
||
An AI-powered analytics engine designed to process financial news articles and extract meaningful insights, sentiment analysis, and market indicators from collected content.
|
||
|
||
## Overview
|
||
|
||
The AI Processor is the intelligent component of the system that analyzes the financial news articles collected by the scraper. It uses natural language processing techniques and machine learning models to extract key insights, determine sentiment, identify market trends, and generate actionable analytics for traders and investors.
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
ai_processor/
|
||
├── app.py # Main AI processing application
|
||
├── config.py # Configuration settings
|
||
├── requirements.txt # Python dependencies
|
||
├── Dockerfile # Docker configuration
|
||
├── README.md # This file
|
||
├── models/ # Machine learning models and NLP components
|
||
│ ├── __init__.py
|
||
│ ├── sentiment_analyzer.py # Sentiment analysis module
|
||
│ ├── topic_classifier.py # News categorization module
|
||
│ └── entity_extractor.py # Named entity recognition
|
||
├── processors/ # Article processing pipelines
|
||
│ ├── __init__.py
|
||
│ ├── text_processor.py # Text cleaning and preprocessing
|
||
│ └── analysis_pipeline.py # Full analysis pipeline
|
||
└── data/ # Processed data storage
|
||
├── insights/
|
||
└── reports/
|
||
```
|
||
|
||
## Features
|
||
|
||
- **Sentiment Analysis**: Determine positive, negative, or neutral sentiment of news articles
|
||
- **Topic Classification**: Categorize articles by financial topics (economics, politics, technology, etc.)
|
||
- **Entity Extraction**: Identify key entities mentioned in articles (stocks, companies, people, organizations)
|
||
- **Market Indicator Detection**: Extract quantitative indicators that may affect stock prices
|
||
- **Insight Generation**: Automated generation of actionable intelligence from news content
|
||
- **Batch Processing**: Process large volumes of articles efficiently
|
||
|
||
## Endpoints
|
||
|
||
### Article Analysis
|
||
- POST `/api/analyze/article` - Analyze a single article for insights
|
||
- POST `/api/analyze/batch` - Process multiple articles in batch mode
|
||
- GET `/api/analyze/status/{task_id}` - Check processing status
|
||
|
||
### Data Access
|
||
- GET `/api/insights/latest` - Get latest analysis insights
|
||
- GET `/api/insights/articles/{article_path}` - Get insights for specific article
|
||
- GET `/api/reports/generate` - Generate comprehensive market analysis report
|
||
|
||
### Model Management
|
||
- GET `/api/models` - List available AI models
|
||
- POST `/api/models/update` - Update or retrain models with new data
|
||
|
||
## Configuration
|
||
|
||
### Environment Variables
|
||
|
||
The AI Processor supports configuration through environment variables:
|
||
|
||
- `MODEL_PATH` - Path to pre-trained AI models (default: `models/`)
|
||
- `ARTICLE_DIR` - Directory containing articles to process (default: `../scraper/articles`)
|
||
- `ENABLE_CACHING` - Enable/disable result caching (default: True)
|
||
- `LOG_LEVEL` - Logging level (DEBUG, INFO, WARNING, ERROR)
|
||
- `MAX_WORKERS` - Number of concurrent processing threads (default: 4)
|
||
|
||
## Installation
|
||
|
||
1. Install dependencies:
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
2. Set up environment variables (optional but recommended):
|
||
```bash
|
||
export MODEL_PATH="/path/to/models"
|
||
export ARTICLE_DIR="/path/to/articles"
|
||
export ENABLE_CACHING=true
|
||
```
|
||
|
||
3. Run the AI processor:
|
||
```bash
|
||
python app.py
|
||
```
|
||
|
||
## Usage Examples
|
||
|
||
### Analyze a single article:
|
||
```bash
|
||
curl -X POST "http://localhost:5001/api/analyze/article" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"path":"/path/to/article.txt","source":"Reuters – Business News"}'
|
||
```
|
||
|
||
### Batch process articles:
|
||
```bash
|
||
curl -X POST "http://localhost:5001/api/analyze/batch" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"article_paths":["/path/to/article1.txt","/path/to/article2.txt"],"include_sentiment":true}'
|
||
```
|
||
|
||
### Get latest insights:
|
||
```bash
|
||
curl "http://localhost:5001/api/insights/latest?limit=10"
|
||
```
|
||
|
||
## Requirements
|
||
|
||
- Python 3.6+
|
||
- NLP libraries (spaCy, NLTK, transformers)
|
||
- Machine learning frameworks (scikit-learn, tensorflow/PyTorch)
|
||
- Additional dependencies listed in `requirements.txt`
|
||
|
||
## License
|
||
|
||
This project is licensed under the MIT License. |