user/jarian/addingScraperDocker (#2)

This fully implements the Stock Docs Project with full Docker Containerization support. This is a working prototype that is actively running on the Media Server. There's a few issues noted, including the following:
- Support for some sites could be improved. Reuters has many articles behind an adblock and some websites present banners that don't need to be processed by our AI engine
- Some caching could be smarter. As the size of files grows, it will get expensive to search through all files to be sure we've not scraped it, ai proccessed it or embedded it.
- Logging could be improved to be much better than just print statements and telemetry could be sent for dashboard monitoring if this were ever to become a full service where we cared about reliability.
- MCP server has been noted to return some poorly matching results. Would be better if it returned nothing at all. And should never really return banners or ads as that provides awful input for the model. Perhaps the model could be told to not care about this, but it's better to just never show irrelevant info to the model

I think this is an overall really good jumping off point, and we've already gotten to see the max capabilities of our system so far. It's a major win to have the Scraper for instance running at all times getting articles from across the web. I look forward to expending this scraper in the near future for projects like scraping all local news websites in the US or general scraping and monitoring of websites.

Co-authored-by: Jarian Cottingham <jariancottingham@dev-machine.local>
Co-authored-by: jarianc <user@example.com>
Reviewed-on: http://git.example.com/jarianc/StockDocs/pulls/2

.gitignore

@@ -16,3 +16,6 @@ scraper/articles/*
 
 # Anything from AI Process
 ai_processor/output/*
+
+# Anything from running processes in the background
+nohup.out

MCPServer/docker-compose.yml

@@ -1,20 +0,0 @@
-services:
-  flask-app:
-    build: .
-    container_name: stockdocs-mcp
-    networks:
-      - ainetwork
-    ports:
-      - "5005:5005"
-    environment:
-      - FLASK_ENV=development
-    deploy:
-      replicas: 1 # You can scale the number of instances (replicas) as needed
-      resources:
-        limits:
-          cpus: '1'
-          memory: 1024M
-networks:
-  ainetwork:
-    external: true
-    name: ainetwork

ai_processor/ai_processor.py

@@ -1,8 +1,10 @@
 import os
 import requests
 import json
+import datetime
+import time
 
-LOCAL_AI_SERVICE_URL = "http://192.168.8.124:11434"
+LOCAL_AI_SERVICE_URL = os.getenv("AI_SERVICE_URL") 
 
 class ArticleFile:
     def __init__(self, filename, content, source="Unfiltered"):
@@ -18,9 +20,21 @@ def load_articles_from_folder(folder_path):
     Loads all articles from text files in the specified folder and returns them as a list of ArticleFile objects.
     """
     articles = []
+    # Skip non-directory files in the root folder
     for newspaper in os.listdir(folder_path):
-        for filename in os.listdir(os.path.join(folder_path, newspaper)):
+        newspaper_path = os.path.join(folder_path, newspaper)
-            file_path = os.path.join(folder_path, newspaper, filename)
+        if not os.path.isdir(newspaper_path):
+            continue
+            
+        for filename in os.listdir(newspaper_path):
+            file_path = os.path.join(newspaper_path, filename)
+            output_path = os.path.join("output", f"{filename}.json")
+            
+            # Skip if already processed
+            if os.path.isfile(output_path):
+                print(f"Skipping already processed article: {filename}")
+                continue
+                
             if os.path.isfile(file_path):
                 with open(file_path, 'r', encoding='utf-8') as f:
                     first_line = f.readline()
@@ -32,7 +46,7 @@ def load_articles_from_folder(folder_path):
                         content = first_line + f.read()
                     articles.append(ArticleFile(filename, content, source))
     
-    print(f"Loaded {len(articles)} articles from folder {folder_path}.")
+    print(f"Loaded {len(articles)} new articles from folder {folder_path}.")
     return articles
 
 def call_local_ai_service(article):
@@ -54,8 +68,11 @@ def call_local_ai_service(article):
     payload = {
         "model": "llama3.2:latest",  
         "prompt": f"{prompt}\n\nArticle:\n{article}",
-        "stream": False
+        "stream": False,
-        # TODO: In the future, we should force the response to be json and adjust the temperature
+        "format": "json",
+        "options": {
+            "temperature": 0.1
+        }
     }
 
     print(f"Sending request to local AI service ...")
@@ -85,7 +102,7 @@ def process_article(article_file):
     processed_article = None
 
     # Check if we already processed this article
-    # TODO : protentially expensive, consider using a database or cache
+    # TODO : potentially expensive, consider using a database or cache
     if os.path.exists(f"output/{article_file.filename}.json"):
         print(f"Article {article_file.filename} already processed, skipping.")
         
@@ -96,6 +113,10 @@ def process_article(article_file):
 
     for attempt in range(3):
         try:
+            if not article_file.content.strip() or len(article_file.content) < 30:
+                print(f"Skipping empty article or article with insufficient content: {article_file.filename}")
+                return None
+
             # Pass the article content to the AI service
             processed_article = call_local_ai_service(article_file.content)
             print(f"Processed article: {processed_article['summary'][:100]}...")
@@ -114,19 +135,35 @@ def process_article(article_file):
 
     return processed_article
 
-# Retrieve the current archive of pulled articles
+while True:
-articles_folder = os.path.join(os.path.dirname(__file__), "../scraper/articles")
+    print("=========================================")
-print("Loading articles from folder " + articles_folder + " ...")
-articles = load_articles_from_folder(articles_folder)
 
-# Process the articles retrieved
+    # Retrieve the current archive of pulled articles
-results = []
+    articles_folder = os.path.join("/app/articles")
-for article in articles: 
+    if not os.path.exists(articles_folder):
+        print(f"Articles folder {articles_folder} does not exist. Please check the path.")
+        exit(1)
+    print("Loading articles from folder " + articles_folder + " ...")
+    articles = load_articles_from_folder(articles_folder)
+
+    # Process the articles retrieved
+    results = []
+    failedProcessCount = 0
+    for article in articles: 
         try:
             result = process_article(article)
 
+            if result is None:
+                failedProcessCount += 1
+                #print(f"Skipping article {article.filename} due to processing error.")
+                continue
+                
             # Save the processed result to a JSON file
             with open("output/" + result['filename'] + ".json", "w", encoding="utf-8") as f:
                 json.dump(result, f, ensure_ascii=False, indent=2)
         except Exception as e:
             print(f"Error processing or saving article {article.filename}: {e}")
+
+    print(f"Processed {len(articles) - failedProcessCount} articles successfully, {failedProcessCount} failed to process.") 
+
+    time.delay(60 * 5)  # Wait for 5 minutes before the next iteration

ai_processor/dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the rest of the application
+COPY . .
+
+# Create output directory
+RUN mkdir -p output
+
+# Command to run the processor
+CMD ["python", "ai_processor.py"]

ai_processor/pyvenv.cfg

@@ -1,5 +1,3 @@
-home = /usr/bin
+home = /Library/Developer/CommandLineTools/usr/bin
 include-system-site-packages = false
-version = 3.12.3
+version = 3.9.6
-executable = /usr/bin/python3.12
-command = /usr/bin/python3 -m venv /home/user/embedding

ai_processor/requirements.txt

@@ -0,0 +1,5 @@
+certifi==2025.7.14
+charset-normalizer==3.4.2
+idna==3.10
+requests==2.32.4
+urllib3==2.5.0

docker-compose.yml

@@ -0,0 +1,49 @@
+services:
+  scraper:
+    build: ./scraper
+    platform: linux/amd64
+    container_name: stockdocs-scraper
+    networks:
+      - ainetwork
+    volumes:
+      - ./scraper/articles:/app/articles
+    environment:
+      - FEED_FILE=rss_feeds.json
+  flask-app:
+    build: ./MCPServer
+    platform: linux/amd64
+    container_name: stockdocs-mcp
+    networks:
+      - ainetwork
+    ports:
+      - "5005:5005"
+    environment:
+      - FLASK_ENV=development
+  ai_processor:
+    build: ./ai_processor
+    platform: linux/amd64
+    container_name: stockdocs-ai-processor
+    networks:
+      - ainetwork
+    volumes:
+      - ./scraper/articles:/app/articles
+      - ./ai_processor/output:/app/output
+    environment:
+      - AI_SERVICE_URL=http://192.168.8.124:11434  # Local AI service IP
+  embedder:
+    build: ./embedding
+    platform: linux/amd64
+    container_name: stockdocs-embedder
+    networks:
+      - ainetwork
+    volumes:
+      - ./ai_processor/output:/app/output
+    environment:
+      - CHROMADB_HOST=chromadb
+      - CHROMADB_PORT=8000
+  
+
+networks:
+  ainetwork:
+    external: true
+    name: ainetwork

embedding/dockerfile

@@ -0,0 +1,22 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Create directories
+RUN mkdir -p input output
+
+# Copy the rest of the application
+COPY . .
+
+# Command to run the embedder
+CMD ["python", "embedder.py"]

embedding/embedder.py

@@ -2,9 +2,13 @@ import os
 import json
 import chromadb
 import uuid
-
+import time
+import datetime
 from sentence_transformers import SentenceTransformer
 
+CHROMADB_HOST = os.getenv("CHROMADB_HOST")
+CHROMADB_PORT = os.getenv("CHROMADB_PORT")
+
 model = SentenceTransformer(
     "Snowflake/snowflake-arctic-embed-m-long",
     device="cpu",               # <-- the only line that changes
@@ -41,18 +45,36 @@ def embed_text(text, specific_context, max_tokens=2048, overlap=256):
     return (chunks, model.encode(chunks, normalize_embeddings=True))
 
 
-client = chromadb.HttpClient(host="localhost", port=8000)
+client = chromadb.HttpClient(host=CHROMADB_HOST, port=CHROMADB_PORT)
 
 # client.delete_collection("news")  # Replace "news" with your collection name
 
 collection = client.get_or_create_collection("news")
 
+# Load cache of previously embedded articles
+cache_file = "embedded_articles_cache.json"
+
 # Load all the processed articles from the output folder
-output_folder = os.path.join(os.path.dirname(__file__), "../ai_processor/output")
+output_folder = "/app/output"
-output_articles = []
+
-# When embedding and upserting, use the source from the processed result
+while True:
-for filename in os.listdir(output_folder):
+    print(f"Starting embedding run at {datetime.datetime.now()}")
+    
+    embedded_cache = {}
+    if os.path.exists(cache_file):
+        with open(cache_file, 'r', encoding='utf-8') as f:
+            embedded_cache = json.load(f)
+
+    output_articles = []
+    processed_articles = []
+    # When embedding and upserting, use the source from the processed result
+    for filename in os.listdir(output_folder):
         if filename.endswith(".json"):
+            # Check if already embedded
+            if filename in embedded_cache:
+                print(f"Article {filename} already embedded, skipping.")
+                continue
+                
             with open(os.path.join(output_folder, filename), 'r', encoding='utf-8') as f:
                 prev_proc = json.load(f)
                 output_articles.append(prev_proc)
@@ -84,7 +106,30 @@ for filename in os.listdir(output_folder):
                         } for d in chunks],  # Metadata for each embedded content
                     )
                     
+                    # Add to processed list for cache update
+                    processed_articles.append(filename)
+                    print(f"Successfully embedded {filename}")
+                    
                 except Exception as e:
                     print(f"Error embedding content for {filename}: {e}")
                     continue
 
+    # Update cache with newly processed articles
+    for filename in processed_articles:
+        embedded_cache[filename] = {
+            "embedded_date": str(datetime.datetime.now()),
+            "status": "completed"
+        }
+
+    # Save updated cache
+    try:
+        with open(cache_file, 'w', encoding='utf-8') as f:
+            json.dump(embedded_cache, f, indent=2)
+        print(f"Updated cache with {len(processed_articles)} newly embedded articles.")
+    except Exception as e:
+        print(f"Error saving cache file: {e}")
+
+    print(f"Embedding process completed. Total articles in cache: {len(embedded_cache)}")
+    print("Waiting 5 minutes before next run...")
+    time.sleep(300)  # Wait 5 minutes (300 seconds)
+

embedding/requirements.txt

@@ -1,82 +1,4 @@
-attrs==23.2.0
+chromadb==1.0.13
-Automat==22.10.0
+einops==0.8.1
-Babel==2.10.3
-bcc==0.29.1
-bcrypt==3.2.2
-blinker==1.7.0
-boto3==1.34.46
-botocore==1.34.46
-certifi==2023.11.17
-chardet==5.2.0
-click==8.1.6
-cloud-init==25.1.2
-colorama==0.4.6
-command-not-found==0.3
-configobj==5.0.8
-constantly==23.10.4
-cryptography==41.0.7
-dbus-python==1.3.2
-distro==1.9.0
-distro-info==1.7+build1
-gpg==1.18.0
-h11==0.14.0
-httplib2==0.20.4
-hyperlink==21.0.0
-idna==3.6
-incremental==22.10.0
-Jinja2==3.1.2
-jmespath==1.0.1
-jsonpatch==1.32
-jsonpointer==2.0
-jsonschema==4.10.3
-launchpadlib==1.11.0
-lazr.restfulclient==0.14.6
-lazr.uri==1.0.6
-Markdown==3.5.2
-markdown-it-py==3.0.0
-MarkupSafe==2.1.5
-mdurl==0.1.2
-netaddr==0.8.0
-netifaces==0.11.0
-oauthlib==3.2.2
-packaging==24.0
-pexpect==4.9.0
-ptyprocess==0.7.0
-pyasn1==0.4.8
-pyasn1-modules==0.2.8
-Pygments==2.17.2
-PyGObject==3.48.2
-PyHamcrest==2.1.0
-PyJWT==2.7.0
-pyOpenSSL==23.2.0
-pyparsing==3.1.1
-pyrsistent==0.20.0
-pyserial==3.5
-python-apt==2.7.7+ubuntu4
-python-dateutil==2.8.2
-python-debian==0.1.49+ubuntu2
-python-magic==0.4.27
-pytz==2024.1
-PyYAML==6.0.1
 requests==2.31.0
-rich==13.7.1
+sentence-transformers==5.0.0
-s3transfer==0.10.1
-service-identity==24.1.0
-setuptools==68.1.2
-six==1.16.0
-sos==4.7.2
-ssh-import-id==5.11
-systemd-python==235
-Twisted==24.3.0
-ubuntu-drivers-common==0.0.0
-ubuntu-pro-client==8001
-ufw==0.36.2
-unattended-upgrades==0.1
-urllib3==2.0.7
-uvicorn==0.27.1
-uvloop==0.19.0
-wadllib==1.3.6
-wheel==0.42.0
-wsproto==1.2.0
-xkit==0.0.0
-zope.interface==6.1

scraper/dockerfile

@@ -0,0 +1,20 @@
+# --- Dockerfile.local ---
+FROM python:3.13.5 
+
+# Firefox + GeckoDriver
+RUN apt-get update && apt-get install -y --no-install-recommends \
+        firefox-esr wget ca-certificates gnupg2 \
+ && GECKO=v0.36.0 && \
+    wget -qO- "https://github.com/mozilla/geckodriver/releases/download/${GECKO}/geckodriver-${GECKO}-linux64.tar.gz" \
+    | tar -xz -C /usr/local/bin geckodriver \
+ && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+RUN playwright install
+RUN playwright install-deps
+RUN python3 -m nltk.downloader punkt_tab # Download NLTK data and bake into image
+COPY . .
+
+CMD ["python3", "scraper.py"]

scraper/dockerfile-selenium

@@ -0,0 +1,22 @@
+# Use an official Selenium Firefox standalone as a base image
+FROM selenium/standalone-firefox:latest
+
+USER root
+# Set the working directory in the container
+WORKDIR /app
+
+# Install Python dependencies
+COPY requirements.txt /app/
+RUN pip install --break-system-packages --no-cache-dir -r requirements.txt && \
+    playwright install
+
+# Copy the current directory contents into the container at /app
+COPY . /app
+RUN mkdir -p /app/articles && \
+    chown seluser:seluser /app/articles && \
+    chmod 755 /app/articles
+
+USER seluser                            
+# Run the Flask app
+CMD ["python3", "scraper.py"]
+

scraper/requirements.txt

@@ -38,9 +38,7 @@ tldextract==5.3.0
 tqdm==4.67.1
 trio==0.30.0
 trio-websocket==0.12.2
-typing-extensions==4.14.0
 tzdata==2025.2
-urllib3==2.5.0
 websocket-client==1.8.0
 wsproto==1.2.0
 

scraper/scraper.py

@@ -8,26 +8,31 @@ from selenium.webdriver.firefox.options import Options as FirefoxOptions
 from newspaper.google_news import GoogleNewsSource
 from concurrent.futures import ProcessPoolExecutor
 import nltk
+from nltk.downloader import Downloader
+
+FEED_FILE = os.getenv("FEED_FILE")
 
 # Ensure necessary NLTK resources are downloaded / Needed for selenium + newspaper4k article parsing
-nltk.download('punkt_tab')
+d = Downloader()
+if not d.is_installed('punkt_tab'):
+    nltk.download('punkt_tab')
 
 articles = []
 
-def load_rss_feed_sources():
+def load_rss_feed_sources(feed_file=FEED_FILE):
     """
     Loads the RSS feed sources from a JSON file.
     """
     print("Loading RSS feed sources from rss_feeds.json...")
 
     try:
-        with open("rss_short_feed.json", "r", encoding="utf-8") as f:
+        with open(FEED_FILE, "r", encoding="utf-8") as f:
             return json.load(f)
     except FileNotFoundError:
-        print("rss_feeds.json not found, returning empty list.")
+        print(FEED_FILE + " not found, returning empty list.")
         return []
     except json.JSONDecodeError:
-        print("Error decoding rss_feeds.json, returning empty list.")
+        print("Error decoding " + FEED_FILE + " , returning empty list.")
         return []
 
 def mine_all_articles(rss_feed_sources, limit=None):
@@ -68,9 +73,10 @@ def save_article_to_file(article, filename, source="Unfiltered"):
     """
     Saves the given article text to a file with the specified filename.
     """
-    os.makedirs("articles", exist_ok=True)
+    # articles dir should already be there
+    # os.makedirs("articles", exist_ok=True)
 
-    outputDir = "articles/"+source if source else "articles"
+    outputDir = "articles/"+source
     os.makedirs(outputDir, exist_ok=True) if source else None
 
     # Sanitize filename: use only the last part of the URL or replace slashes
@@ -86,6 +92,12 @@ def get_article_with_selenium(url):
     options = FirefoxOptions()
     options.add_argument("--headless")
     driver = webdriver.Firefox(options=options)
+    driver.set_page_load_timeout(30)  # 30 seconds timeout
+
+
+    #driver = webdriver.Remote(
+    #command_executor='http://localhost:4444/wd/hub',
+    #options=options)
     try:
         driver.get(url)
         time.sleep(5)  # Wait for JS to load
@@ -112,20 +124,6 @@ def get_article_with_playwright(url):
         time.sleep(5)  # Adjust as needed for the page to load completely
 
         html = page.content()
-        '''
-        # Check for iframes and use the first one's content if present
-        frames = page.frames
-        main_frame = page.main_frame
-        for frame in frames:
-            if frame != main_frame:
-                try:
-                    frame.wait_for_load_state("domcontentloaded", timeout=5000)
-                    html = frame.content()
-                    print("Extracted content from iframe.")
-                    break
-                except Exception as e:
-                    print(f"Could not extract iframe content: {e}")
-        '''
 
         browser.close()
 
@@ -135,12 +133,12 @@ def get_article_with_playwright(url):
     return article.text
 
 
-def pull_articles(link, source, title=None, save_to_file=True):
+def pull_article(link, source, title=None, save_to_file=True):
     filename = title if title else link
     safe_filename = generate_filename_from_url(filename)
-    if os.path.exists(os.path.join("articles", safe_filename)):
+    if os.path.exists(os.path.join("articles", source, safe_filename)):
         print(f"Article already cached: {filename}")
-        with open(os.path.join("articles", safe_filename), "r", encoding="utf-8") as f:
+        with open(os.path.join("articles", source, safe_filename), "r", encoding="utf-8") as f:
             return f.read()
         
     text = ""
@@ -148,20 +146,6 @@ def pull_articles(link, source, title=None, save_to_file=True):
     time.sleep(5) # Since we spawn lots of processes, we need to sleep at the start
 
     try:
-        # --- Google News Source handling temporarily disabled ---
-        # if "news.google.com" in link:
-        #     gn = GoogleNewsSource(link)
-        #     gn.build()
-        #     # Pull the first article from the Google News cluster
-        #     if gn.articles:
-        #         article = gn.articles[0]
-        #         article.download()
-        #         article.parse()
-        #         text = article.text
-        #         if not text or len(text) < 200:
-        #             raise ValueError("\tGoogle News article text too short, falling back to Selenium.")
-        #         print(f"\tSuccessfully pulled Google News article from {link}")
-        # else:
         article = newspaper.article(link)
         article.download()
         article.parse()
@@ -196,24 +180,50 @@ def pull_articles(link, source, title=None, save_to_file=True):
         save_article_to_file(text, filename, source)
     return text
 
-# Pull the RSS feed sources from the JSON file
+while True:
-rss_feed_sources = load_rss_feed_sources()
+    print("=========================================")
+    print("Starting new scraping iteration...")
+    # Pull the RSS feed sources from the JSON file
+    rss_feed_sources = load_rss_feed_sources()
 
-# Mine all articles from the RSS feed sources
+    # Mine all articles from the RSS feed sources
-rss_feed_links = mine_all_articles(rss_feed_sources)
+    rss_feed_links = mine_all_articles(rss_feed_sources)
 
-# Randomize the order of the links to help with load balancing
+    # Randomize the order of the links to help with load balancing
-import random
+    import random
-random.shuffle(rss_feed_links)
+    random.shuffle(rss_feed_links)
 
-# Pull the latest article from a variety of news sources
+    link_list = [link for _, title, link in rss_feed_links] 
-#[pull_articles(link, source, title) for source, title, link in rss_feed_links]
+    source_list = [source for source, _, _ in rss_feed_links]
+    title_list = [title for _, title, link in rss_feed_links]
 
-link_list = [link for _, title, link in rss_feed_links] 
+    with ProcessPoolExecutor() as executor:
-source_list = [source for source, _, _ in rss_feed_links]
+        futures = [executor.submit(pull_article, link, source, title) for link, source, title in zip(link_list, source_list, title_list)]
-title_list = [title for _, title, link in rss_feed_links]
+        results = []
+        errors = []
+        for future in futures:
+            try:
+                results.append(future.result(timeout=60))  # seconds
+            except Exception as e:
+                print(f"Error in pull_article: {e}")
+                errors.append(e)
+        print(f"Attempted to Pull {len(results)} articles in parallel.")
+        print(f"Encountered {len(errors)} errors during article pulling. " +\
+            "Outputting errors to a local file.")
         
-with ProcessPoolExecutor() as executor:
+        # Ouput errors to a local file
-    results = list(executor.map(pull_articles, link_list, source_list, title_list))
+        if errors:
-    print(f"Pulled {len(results)} articles in parallel.")
+            with open("errors.txt", "w", encoding="utf-8") as f:
+                for error in errors:
+                    f.write(str(error) + "\n")
+            print(f"Errors logged to errors.txt")
+
+        # Print all results to a log file
+        with open("results.txt", "w", encoding="utf-8") as f:
+            for result in results:
+                f.write(result + "\n")
         print("All articles pulled successfully.")
+
+    # Sleep for a while before the next iteration
+    print("Sleeping for 15 minutes before the next iteration...")
+    time.sleep(15 * 60)  # Sleep for 15 minutes