StockDocs/articleServer/README.md

Article Server

A Flask-based HTTP server that provides access to news articles stored in a directory structure, with time-based filtering and outlet-specific querying capabilities.

Overview

This server allows you to query news articles from various sources based on:

• Time range (last hour, day, week, month)
• Specific news outlets
• Direct content retrieval by file path

Articles are organized in a nested directory structure where each news outlet has its own subdirectory containing the respective articles.

Directory Structure

The server expects the following directory structure:

scraper/
└── articles/
    ├── Reuters – Business News/
    │   ├── article1.txt
    │   ├── article2.txt
    │   └── ...
    ├── Associated Press – Business/
    │   ├── article1.txt
    │   └── ...
    └── ...

Endpoints

1. Get Articles (/articles)

Retrieve articles within a specified time range.

Method: GET Parameters:

• time_range (optional): hour, day, week, month (default: hour)
• outlets (optional): comma-separated list of news outlet names

Example:

# Get articles from last day for specific outlets
curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"

# Get all articles from the last hour
curl "http://localhost:5008/articles"

Response:

{
  "articles": [
    {
      "path": "/path/to/article.txt",
      "name": "article.txt",
      "outlet": "Reuters – Business News",
      "created_at": "2023-10-17T14:30:00"
    }
  ],
  "count": 5,
  "time_range": "hour",
  "outlets": ["Reuters – Business News"]
}

2. Get Article Content (/article/content)

Retrieve the full content of a specific article by file path.

Method: GET Parameters:

• path (required): Absolute path to the article file

Example:

# Get full content of an article
curl "http://localhost:5008/article/content?path=/full/path/to/article.txt"

Response:

{
  "path": "/full/path/to/article.txt",
  "name": "article.txt",
  "outlet": "Reuters – Business News",
  "content": "Full article content here..."
}

3. Get Available Outlets (/outlets)

List all available news outlets.

Method: GET Example:

curl "http://localhost:5008/outlets"

Response:

{
  "news_outlets": [
    "Reuters – Business News",
    "Associated Press – Business",
    "Financial Times",
    ...
  ],
  "count": 60
}

4. Health Check (/health)

Simple health check endpoint.

Method: GET Example:

curl "http://localhost:5008/health"

Response:

{
  "status": "healthy"
}

Configuration

Environment Variables

• ARTICLE_DIR (optional): Path to the article directory. Defaults to scraper/articles if not set.

Installation

1. Install dependencies:

pip install -r requirements.txt

2. Set the article directory path (optional):

export ARTICLE_DIR="/path/to/your/articles"

3. Run the server:

python run_server.py

Or with Docker:

docker build -t article-server .
docker run -p 5008:5008 article-server

Usage Examples

Get recent articles from all outlets:

curl "http://localhost:5008/articles?time_range=hour"

Get articles from the last day for specific outlets:

curl "http://localhost:5008/articles?time_range=day&outlets=Reuters – Business News,Associated Press – Business"

Get all available news outlets:

curl "http://localhost:5008/outlets"

Retrieve full content of a specific article:

curl "http://localhost:5008/article/content?path=/absolute/path/to/your/article.txt"

Security Notes

• The server validates that all requested file paths are within the configured article directory to prevent directory traversal attacks
• All file paths must be absolute and within the allowed directory structure
• Path parameters are URL decoded for proper handling of special characters

Requirements

• Python 3.6+
• Flask 2.3.3

License

This project is licensed under the MIT License.