Merge pull request 'fix: repair middleware wiring, input validation, and resource lifecycle' (#17) from improve/v1 into main
Some checks are pending
CI / lint (push) Waiting to run
CI / test (push) Waiting to run
CI / docker-build (push) Waiting to run
CI / security (push) Waiting to run
CI / build-result (push) Blocked by required conditions

Reviewed-on: https://git.example.com/jarianc/google-mcp/pulls/17
This commit is contained in:
Jarian Cottingham 2026-08-20 18:37:33 -05:00
commit fdc67d5887
29 changed files with 391 additions and 94 deletions

11
.dockerignore Normal file
View File

@ -0,0 +1,11 @@
.git
__pycache__
*.py[cod]
.pytest_cache
.env
tests
test_client.py
integration_test.py
docs
*.md
docker-compose.yml

7
.gitignore vendored Normal file
View File

@ -0,0 +1,7 @@
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.env
.venv/
venv/

21
LICENSE Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Jarian Cottingham
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

134
README.md Normal file
View File

@ -0,0 +1,134 @@
# MCP Search Servers
Model Context Protocol (MCP) servers that expose **Google** and **DuckDuckGo** search
as tools for LLM clients (Claude Desktop, Cursor, VS Code, ...), backed by a
self-hosted [SearXNG](https://github.com/searxng/searxng) metasearch instance.
SearXNG aggregates results from Google, DuckDuckGo, Brave, Wikipedia, and more with
its own request handling, so a single SearXNG container serves both MCP servers —
no browser fingerprinting, no API keys, fully self-hosted.
## Services
| Service | Port | MCP tool | Description |
|-------------------|------|--------------------|--------------------------------------|
| `google-mcp` | 3001 | `google_search` | Google results via SearXNG |
| `duckduckgo-mcp` | 3002 | `duckduckgo_search`| DuckDuckGo results via SearXNG |
| `searxng` | — | — | Internal metasearch engine (not published) |
Each server exposes:
- `/sse` + `/messages/` — MCP SSE transport
- `/search?q=...&num=N` — plain HTTP JSON endpoint
- `/health` — liveness probe
## Quick Start
```bash
cp .env.example .env # fill in SEARXNG_SECRET
docker compose up -d --build
```
Generate a SearXNG secret:
```bash
python3 -c "import secrets; print(secrets.token_hex(32))"
```
Verify:
```bash
curl http://localhost:3001/health
curl "http://localhost:3001/search?q=python&num=3"
```
Then point your LLM client at `http://localhost:3001/sse` (Google) and/or
`http://localhost:3002/sse` (DuckDuckGo). Client configuration examples for
Claude Desktop, Cursor/Windsurf, and VS Code are in [USAGE.md](USAGE.md).
## Tool Contract
```json
{
"name": "google_search",
"arguments": { "query": "Python programming language", "num_results": 5 }
}
```
Returns numbered results with title, URL, and snippet (max 20 results,
queries capped at 500 characters).
## Configuration
| Variable | Default | Description |
|----------------------|----------------------|------------------------------------------|
| `SEARXNG_SECRET` | — (required) | SearXNG session/CSRF secret |
| `RATE_LIMIT_SECONDS` | 5 / 3 | Minimum seconds between search requests |
| `MCP_PORT` | 3001 / 3002 | Port the MCP server listens on |
| `SEARXNG_URL` | `http://searxng:8080`| URL of the SearXNG instance |
## Security Model
- **Internal network** — SearXNG is on a `internal: true` Docker network; only the
two MCP containers can reach it. It is not published to the host.
- **Per-IP rate limiting** — HTTP endpoints allow 30 requests/minute per IP
(`/health` exempt); excess requests get `429`.
- **Request pacing** — each server also enforces a minimum interval between
upstream SearXNG queries to avoid throttling.
- **No auth on `/search` and `/sse`** — the servers assume they are bound to a
trusted network. Expose them publicly only behind an auth proxy.
- **Input validation** — query length capped, result counts clamped to 1..20,
malformed parameters return `400` instead of crashing.
## Optional: Playwright Scraping Fallback
`lib/google_search.py` and `lib/duckduckgo_search.py` contain a legacy
Playwright-based scraping path (single shared browser, stealth user agent,
HTML parsing with multiple fallback strategies). It is **not** used by the
production servers — Google aggressively blocks headless browsers and SearXNG
is the working path. If you want to run the scraper directly:
```bash
pip install playwright
playwright install chromium
```
## Development
```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
ruff check .
pytest tests/ -v
# End-to-end test (requires running containers)
python3 integration_test.py
```
## Project Structure
```
├── docker-compose.yml # All 3 services
├── searxng-settings.yml # SearXNG engine config
├── google-mcp/
│ ├── Dockerfile
│ └── server.py # Google MCP server (SearXNG-backed)
├── duckduckgo-mcp/
│ ├── Dockerfile
│ └── server.py # DuckDuckGo MCP server (SearXNG-backed)
├── lib/
│ ├── playwright_manager.py # Single-instance browser manager (optional path)
│ ├── rate_limiter.py # Async minimum-interval rate limiter
│ ├── google_search.py # Playwright Google scraper (optional path)
│ └── duckduckgo_search.py # Playwright DuckDuckGo scraper (optional path)
├── tests/
│ ├── test_rate_limiter.py
│ └── test_search_parsing.py
├── integration_test.py # End-to-end MCP SSE test
└── test_client.py # Manual stdio/SSE smoke client
```
## License
[MIT](LICENSE)

View File

@ -5,7 +5,7 @@ MCP (Model Context Protocol) servers that expose Google and DuckDuckGo search as
## Quick Start ## Quick Start
```bash ```bash
cd google-mcp # From the repository root
docker compose up -d --build docker compose up -d --build
``` ```
@ -169,9 +169,11 @@ python3 integration_test.py
## Troubleshooting ## Troubleshooting
**No results returned** — SearXNG may be blocked on your network. Check which engines are working: **No results returned** — SearXNG may be blocked on your network. Check which engines are working
(SearXNG is on an internal Docker network, so query it from inside its container):
```bash ```bash
curl "http://localhost:8080/search?q=test&format=json" | python3 -m json.tool docker compose exec searxng python -c \
"import urllib.request; print(urllib.request.urlopen('http://localhost:8080/search?q=test&format=json').read()[:500])"
``` ```
**Services not starting** — Check for port conflicts: **Services not starting** — Check for port conflicts:

2
conftest.py Normal file
View File

@ -0,0 +1,2 @@
# Ensures the repo root is on sys.path so `lib` and the server packages
# are importable in tests without installation.

View File

@ -16,7 +16,7 @@ services:
networks: networks:
- mcp-internal - mcp-internal
healthcheck: healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"] test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3

View File

@ -1,5 +1,4 @@
import asyncio import asyncio
import json
import logging import logging
import os import os
import signal import signal
@ -9,15 +8,15 @@ from contextlib import asynccontextmanager
from datetime import datetime, timezone from datetime import datetime, timezone
import httpx import httpx
from mcp.server import Server from mcp.server import Server
from mcp.server.sse import SseServerTransport from mcp.server.sse import SseServerTransport
from mcp.types import TextContent, Tool
from starlette.applications import Starlette from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request from starlette.requests import Request
from starlette.responses import JSONResponse, Response from starlette.responses import JSONResponse, Response
from starlette.routing import Mount, Route from starlette.routing import Mount, Route
from mcp.types import Tool, TextContent
logging.basicConfig( logging.basicConfig(
level=logging.INFO, level=logging.INFO,
@ -31,6 +30,19 @@ PORT = int(os.environ.get("MCP_PORT", "3002"))
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080") SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "3")) RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "3"))
DEFAULT_RESULTS = 10
MAX_RESULTS = 20
MAX_QUERY_LEN = 500
def parse_num_results(value) -> int:
"""Parse a result-count value, clamped to 1..MAX_RESULTS."""
try:
num = int(value)
except (TypeError, ValueError):
return DEFAULT_RESULTS
return max(1, min(num, MAX_RESULTS))
# --- HTTPX singleton (#8) --- # --- HTTPX singleton (#8) ---
@ -105,10 +117,17 @@ class RateLimitMiddleware(BaseHTTPMiddleware):
client_ip = request.client.host if request.client else "unknown" client_ip = request.client.host if request.client else "unknown"
now = datetime.now(timezone.utc).timestamp() now = datetime.now(timezone.utc).timestamp()
cutoff = now - self.window_seconds
async with self._lock: async with self._lock:
if len(self._requests) > 1000:
for ip in [
ip
for ip, ts in self._requests.items()
if not ts or ts[-1] <= cutoff
]:
del self._requests[ip]
timestamps = self._requests[client_ip] timestamps = self._requests[client_ip]
cutoff = now - self.window_seconds
self._requests[client_ip] = [t for t in timestamps if t > cutoff] self._requests[client_ip] = [t for t in timestamps if t > cutoff]
if len(self._requests[client_ip]) >= self.max_requests: if len(self._requests[client_ip]) >= self.max_requests:
return JSONResponse( return JSONResponse(
@ -155,15 +174,20 @@ async def do_search(query: str, num_results: int, engine: str):
# --- HTTP endpoint --- # --- HTTP endpoint ---
async def search_http(request: Request): async def search_http(request: Request):
query = request.query_params.get("q", "") query = request.query_params.get("q", "").strip()
if not query: if not query:
return JSONResponse({"error": "'q' parameter required"}, status_code=400) return JSONResponse({"error": "'q' parameter required"}, status_code=400)
if len(query) > MAX_QUERY_LEN:
return JSONResponse(
{"error": f"'q' must be {MAX_QUERY_LEN} characters or fewer"},
status_code=400,
)
num = min(int(request.query_params.get("num", "10")), 20) num = parse_num_results(request.query_params.get("num"))
try: try:
results = await do_search(query, num, "duckduckgo") results = await do_search(query, num, "duckduckgo")
return JSONResponse({"query": query, "results": results}) return JSONResponse({"query": query, "results": results})
except Exception as e: except Exception:
logger.exception("Search failed") logger.exception("Search failed")
return JSONResponse({"error": "Search service unavailable"}, status_code=503) return JSONResponse({"error": "Search service unavailable"}, status_code=503)
@ -204,14 +228,21 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
raise ValueError(f"Unknown tool: {name}") raise ValueError(f"Unknown tool: {name}")
query = arguments.get("query", "") query = arguments.get("query", "")
if not query: if not isinstance(query, str) or not query:
return [TextContent(type="text", text="Error: 'query' is required.")] return [TextContent(type="text", text="Error: 'query' is required.")]
if len(query) > MAX_QUERY_LEN:
return [
TextContent(
type="text",
text=f"Error: query must be {MAX_QUERY_LEN} characters or fewer.",
)
]
num_results = min(int(arguments.get("num_results", 10)), 20) num_results = parse_num_results(arguments.get("num_results"))
try: try:
results = await do_search(query, num_results, "duckduckgo") results = await do_search(query, num_results, "duckduckgo")
except Exception as e: except Exception:
logger.exception("Search failed") logger.exception("Search failed")
return [TextContent(type="text", text="Search failed: service unavailable")] return [TextContent(type="text", text="Search failed: service unavailable")]
@ -248,13 +279,14 @@ async def handle_sse(request):
starlette_app = Starlette( starlette_app = Starlette(
debug=False, debug=False,
lifespan=lifespan,
routes=[ routes=[
Route("/health", endpoint=health_check), Route("/health", endpoint=health_check),
Route("/search", endpoint=search_http), Route("/search", endpoint=search_http),
Route("/sse", endpoint=handle_sse), Route("/sse", endpoint=handle_sse),
Mount("/messages/", app=sse.handle_post_message), Mount("/messages/", app=sse.handle_post_message),
], ],
middleware=[RateLimitMiddleware], middleware=[Middleware(RateLimitMiddleware)],
) )

View File

@ -1,5 +1,4 @@
import asyncio import asyncio
import json
import logging import logging
import os import os
import signal import signal
@ -9,15 +8,15 @@ from contextlib import asynccontextmanager
from datetime import datetime, timezone from datetime import datetime, timezone
import httpx import httpx
from mcp.server import Server from mcp.server import Server
from mcp.server.sse import SseServerTransport from mcp.server.sse import SseServerTransport
from mcp.types import TextContent, Tool
from starlette.applications import Starlette from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request from starlette.requests import Request
from starlette.responses import JSONResponse, Response from starlette.responses import JSONResponse, Response
from starlette.routing import Mount, Route from starlette.routing import Mount, Route
from mcp.types import Tool, TextContent
logging.basicConfig( logging.basicConfig(
level=logging.INFO, level=logging.INFO,
@ -31,6 +30,19 @@ PORT = int(os.environ.get("MCP_PORT", "3001"))
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080") SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "5")) RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "5"))
DEFAULT_RESULTS = 10
MAX_RESULTS = 20
MAX_QUERY_LEN = 500
def parse_num_results(value) -> int:
"""Parse a result-count value, clamped to 1..MAX_RESULTS."""
try:
num = int(value)
except (TypeError, ValueError):
return DEFAULT_RESULTS
return max(1, min(num, MAX_RESULTS))
# --- HTTPX singleton (#8) --- # --- HTTPX singleton (#8) ---
@ -105,10 +117,17 @@ class RateLimitMiddleware(BaseHTTPMiddleware):
client_ip = request.client.host if request.client else "unknown" client_ip = request.client.host if request.client else "unknown"
now = datetime.now(timezone.utc).timestamp() now = datetime.now(timezone.utc).timestamp()
cutoff = now - self.window_seconds
async with self._lock: async with self._lock:
if len(self._requests) > 1000:
for ip in [
ip
for ip, ts in self._requests.items()
if not ts or ts[-1] <= cutoff
]:
del self._requests[ip]
timestamps = self._requests[client_ip] timestamps = self._requests[client_ip]
cutoff = now - self.window_seconds
self._requests[client_ip] = [t for t in timestamps if t > cutoff] self._requests[client_ip] = [t for t in timestamps if t > cutoff]
if len(self._requests[client_ip]) >= self.max_requests: if len(self._requests[client_ip]) >= self.max_requests:
return JSONResponse( return JSONResponse(
@ -155,15 +174,20 @@ async def do_search(query: str, num_results: int, engine: str):
# --- HTTP endpoint --- # --- HTTP endpoint ---
async def search_http(request: Request): async def search_http(request: Request):
query = request.query_params.get("q", "") query = request.query_params.get("q", "").strip()
if not query: if not query:
return JSONResponse({"error": "'q' parameter required"}, status_code=400) return JSONResponse({"error": "'q' parameter required"}, status_code=400)
if len(query) > MAX_QUERY_LEN:
return JSONResponse(
{"error": f"'q' must be {MAX_QUERY_LEN} characters or fewer"},
status_code=400,
)
num = min(int(request.query_params.get("num", "10")), 20) num = parse_num_results(request.query_params.get("num"))
try: try:
results = await do_search(query, num, "google") results = await do_search(query, num, "google")
return JSONResponse({"query": query, "results": results}) return JSONResponse({"query": query, "results": results})
except Exception as e: except Exception:
logger.exception("Search failed") logger.exception("Search failed")
return JSONResponse({"error": "Search service unavailable"}, status_code=503) return JSONResponse({"error": "Search service unavailable"}, status_code=503)
@ -204,14 +228,21 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
raise ValueError(f"Unknown tool: {name}") raise ValueError(f"Unknown tool: {name}")
query = arguments.get("query", "") query = arguments.get("query", "")
if not query: if not isinstance(query, str) or not query:
return [TextContent(type="text", text="Error: 'query' is required.")] return [TextContent(type="text", text="Error: 'query' is required.")]
if len(query) > MAX_QUERY_LEN:
return [
TextContent(
type="text",
text=f"Error: query must be {MAX_QUERY_LEN} characters or fewer.",
)
]
num_results = min(int(arguments.get("num_results", 10)), 20) num_results = parse_num_results(arguments.get("num_results"))
try: try:
results = await do_search(query, num_results, "google") results = await do_search(query, num_results, "google")
except Exception as e: except Exception:
logger.exception("Search failed") logger.exception("Search failed")
return [TextContent(type="text", text="Search failed: service unavailable")] return [TextContent(type="text", text="Search failed: service unavailable")]
@ -248,13 +279,14 @@ async def handle_sse(request):
starlette_app = Starlette( starlette_app = Starlette(
debug=False, debug=False,
lifespan=lifespan,
routes=[ routes=[
Route("/health", endpoint=health_check), Route("/health", endpoint=health_check),
Route("/search", endpoint=search_http), Route("/search", endpoint=search_http),
Route("/sse", endpoint=handle_sse), Route("/sse", endpoint=handle_sse),
Mount("/messages/", app=sse.handle_post_message), Mount("/messages/", app=sse.handle_post_message),
], ],
middleware=[RateLimitMiddleware], middleware=[Middleware(RateLimitMiddleware)],
) )

View File

@ -3,6 +3,7 @@
import asyncio import asyncio
import json import json
import sys import sys
import aiohttp import aiohttp
@ -76,7 +77,7 @@ async def test_server(name, host, port, tool_name, query):
"jsonrpc": "2.0", "id": 1, "method": "initialize", "jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {"protocolVersion": "2024-11-05", "capabilities": {}, "params": {"protocolVersion": "2024-11-05", "capabilities": {},
"clientInfo": {"name": "test", "version": "1.0"}}, "clientInfo": {"name": "test", "version": "1.0"}},
}) as resp: }):
pass pass
init = await collector.wait_for_id(1, timeout=10) init = await collector.wait_for_id(1, timeout=10)
@ -88,25 +89,31 @@ async def test_server(name, host, port, tool_name, query):
# Initialized notification # Initialized notification
async with session.post(messages_endpoint, json={ async with session.post(messages_endpoint, json={
"jsonrpc": "2.0", "method": "notifications/initialized", "jsonrpc": "2.0", "method": "notifications/initialized",
}) as resp: }):
pass pass
# List tools # List tools
async with session.post(messages_endpoint, json={ async with session.post(messages_endpoint, json={
"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}, "jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {},
}) as resp: }):
pass pass
tools = await collector.wait_for_id(2, timeout=10) tools = await collector.wait_for_id(2, timeout=10)
if tools: if tools:
print(f" Tools: {[t['name'] for t in tools.get('result', {}).get('tools', [])]}") tool_names = [
t["name"] for t in tools.get("result", {}).get("tools", [])
]
print(f" Tools: {tool_names}")
# Call search # Call search
print(f" Searching: '{query}'") print(f" Searching: '{query}'")
async with session.post(messages_endpoint, json={ async with session.post(messages_endpoint, json={
"jsonrpc": "2.0", "id": 3, "method": "tools/call", "jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {"name": tool_name, "arguments": {"query": query, "num_results": 5}}, "params": {
}) as resp: "name": tool_name,
"arguments": {"query": query, "num_results": 5},
},
}):
pass pass
result = await collector.wait_for_id(3, timeout=30) result = await collector.wait_for_id(3, timeout=30)
@ -145,11 +152,14 @@ async def main():
print("Waiting for services to start...") print("Waiting for services to start...")
await asyncio.sleep(5) await asyncio.sleep(5)
results["google"] = await test_server("Google", host, 3001, "google_search", "Python programming language") query = "Python programming language"
results["google"] = await test_server("Google", host, 3001, "google_search", query)
await asyncio.sleep(2) await asyncio.sleep(2)
results["duckduckgo"] = await test_server("DuckDuckGo", host, 3002, "duckduckgo_search", "Python programming language") results["duckduckgo"] = await test_server(
"DuckDuckGo", host, 3002, "duckduckgo_search", query
)
print(f"\n{'='*60}") print(f"\n{'='*60}")
print("SUMMARY") print("SUMMARY")

View File

@ -1,11 +1,25 @@
from lib.playwright_manager import PlaywrightManager """Shared search utilities.
from lib.rate_limiter import RateLimiter
from lib.google_search import GoogleSearch
from lib.duckduckgo_search import DuckDuckGoSearch
__all__ = [ Playwright-based scrapers are imported lazily so the package (and its
"PlaywrightManager", rate limiter) works without the optional ``playwright`` dependency.
"RateLimiter", """
"GoogleSearch",
"DuckDuckGoSearch", from lib.rate_limiter import RateLimiter
]
__all__ = ["PlaywrightManager", "RateLimiter", "GoogleSearch", "DuckDuckGoSearch"]
def __getattr__(name):
if name == "PlaywrightManager":
from lib.playwright_manager import PlaywrightManager
return PlaywrightManager
if name == "GoogleSearch":
from lib.google_search import GoogleSearch
return GoogleSearch
if name == "DuckDuckGoSearch":
from lib.duckduckgo_search import DuckDuckGoSearch
return DuckDuckGoSearch
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

View File

@ -1,6 +1,6 @@
import asyncio import asyncio
import logging import logging
from typing import List, Dict, Any, Optional from typing import Any, Dict, List
from lib.playwright_manager import PlaywrightManager from lib.playwright_manager import PlaywrightManager
from lib.rate_limiter import RateLimiter from lib.rate_limiter import RateLimiter

View File

@ -1,7 +1,8 @@
import asyncio import asyncio
import logging import logging
import re import re
from typing import List, Dict, Any import urllib.parse
from typing import Any, Dict, List
from lib.playwright_manager import PlaywrightManager from lib.playwright_manager import PlaywrightManager
from lib.rate_limiter import RateLimiter from lib.rate_limiter import RateLimiter
@ -29,7 +30,10 @@ class GoogleSearch:
try: try:
page = await PlaywrightManager.get_page(stealth=True) page = await PlaywrightManager.get_page(stealth=True)
url_params = f"{GOOGLE_URL}?q={query}&num={min(num_results, 20)}&hl={language}" url_params = (
f"{GOOGLE_URL}?q={urllib.parse.quote(query)}"
f"&num={min(num_results, 20)}&hl={language}"
)
logger.info(f"Navigating to Google search: {query}") logger.info(f"Navigating to Google search: {query}")
await page.goto(url_params, wait_until="domcontentloaded", timeout=30000) await page.goto(url_params, wait_until="domcontentloaded", timeout=30000)
@ -98,8 +102,6 @@ class GoogleSearch:
def _clean_url(self, raw_url: str) -> str: def _clean_url(self, raw_url: str) -> str:
"""Decode Google redirect URL.""" """Decode Google redirect URL."""
import urllib.parse
raw_url = urllib.parse.unquote(raw_url) raw_url = urllib.parse.unquote(raw_url)
# Remove tracking parameters # Remove tracking parameters
url = re.sub(r"&[a-z_]+=.*$", "", raw_url) url = re.sub(r"&[a-z_]+=.*$", "", raw_url)
@ -118,7 +120,13 @@ class GoogleSearch:
for i, result in enumerate(results): for i, result in enumerate(results):
for block in snippet_blocks: for block in snippet_blocks:
clean = re.sub(r'<[^>]+>', "", block).strip() clean = re.sub(r'<[^>]+>', "", block).strip()
if 50 < len(clean) < 300 and result["url"] in html[max(0, html.find(clean) - 500):html.find(clean) + 500]: if not 50 < len(clean) < 300:
continue
pos = html.find(clean)
if pos == -1:
continue
window = html[max(0, pos - 500): pos + 500]
if result["url"] in window:
result["snippet"] = clean[:200] result["snippet"] = clean[:200]
break break
return results return results

View File

@ -2,7 +2,7 @@ import asyncio
import logging import logging
from typing import Optional from typing import Optional
from playwright.async_api import async_playwright, Browser, BrowserContext, Page from playwright.async_api import Browser, BrowserContext, Page, async_playwright
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -75,7 +75,7 @@ class PlaywrightManager:
stealth = Stealth() stealth = Stealth()
await stealth.apply_stealth_async(page) await stealth.apply_stealth_async(page)
logger.debug("Stealth mode applied to page") logger.debug("Stealth mode applied to page")
except (ImportError, Exception) as e: except Exception as e:
logger.warning(f"Could not apply stealth: {e}") logger.warning(f"Could not apply stealth: {e}")
return page return page

37
pyproject.toml Normal file
View File

@ -0,0 +1,37 @@
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "mcp-search-servers"
version = "1.1.0"
description = "MCP servers exposing Google and DuckDuckGo search via SearXNG"
readme = "README.md"
license = { text = "MIT" }
requires-python = ">=3.10"
dependencies = [
"mcp>=1.0.0,<2.0.0",
"httpx>=0.27.0",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
"ruff>=0.4",
]
playwright = [
"playwright>=1.44",
]
[tool.setuptools]
packages = ["lib"]
[tool.ruff]
line-length = 100
target-version = "py310"
[tool.ruff.lint]
select = ["E", "F", "W", "I"]
[tool.pytest.ini_options]
testpaths = ["tests"]

View File

@ -7,10 +7,8 @@ Usage:
""" """
import asyncio import asyncio
import json
import subprocess import subprocess
import sys import sys
import time
async def test_mcp_server(service_name: str): async def test_mcp_server(service_name: str):
@ -36,7 +34,13 @@ async def test():
await asyncio.sleep(0.5) await asyncio.sleep(0.5)
init_req = json.loads(await read.__anext__()) init_req = json.loads(await read.__anext__())
# Send initialize response # Send initialize response
init_result = {{'jsonrpc': '2.0', 'id': init_req['id'], 'result': {{'protocolVersion': '2024-11-05', 'capabilities': {{}}, 'serverInfo': {{'name': 'test', 'version': '1.0'}}}}}} init_result = {{
'jsonrpc': '2.0', 'id': init_req['id'],
'result': {{
'protocolVersion': '2024-11-05', 'capabilities': {{}},
'serverInfo': {{'name': 'test', 'version': '1.0'}},
}},
}}
await write.send(json.dumps(init_result)) await write.send(json.dumps(init_result))
# Client sends initialized notification # Client sends initialized notification
await read.__anext__() await read.__anext__()
@ -68,7 +72,6 @@ async def test():
async def test_with_curl(service_name: str): async def test_with_curl(service_name: str):
"""Test using docker exec to run a Python test directly in the container.""" """Test using docker exec to run a Python test directly in the container."""
container_name = f"{service_name}-mcp" container_name = f"{service_name}-mcp"
module = service_name.replace("-", "_")
test_script = f""" test_script = f"""
import sys, asyncio, os import sys, asyncio, os

View File

@ -1,19 +1,9 @@
import asyncio import asyncio
import time import time
import pytest
import pytest_asyncio
from lib.rate_limiter import RateLimiter from lib.rate_limiter import RateLimiter
@pytest.fixture
def event_loop():
loop = asyncio.new_event_loop()
yield loop
loop.close()
class TestRateLimiter: class TestRateLimiter:
"""Test rate limiter enforces minimum delays between requests.""" """Test rate limiter enforces minimum delays between requests."""
@ -23,9 +13,9 @@ class TestRateLimiter:
start = time.monotonic() start = time.monotonic()
await limiter.acquire() await limiter.acquire()
elapsed = time.monotonic() - start elapsed = time.monotonic() - start
assert elapsed < 0.1 return elapsed
asyncio.get_event_loop().run_until_complete(run()) assert asyncio.run(run()) < 0.1
def test_enforces_minimum_interval(self): def test_enforces_minimum_interval(self):
async def run(): async def run():
@ -34,28 +24,27 @@ class TestRateLimiter:
start = time.monotonic() start = time.monotonic()
await limiter.acquire() await limiter.acquire()
elapsed = time.monotonic() - start return time.monotonic() - start
assert elapsed >= 0.25 assert asyncio.run(run()) >= 0.25
asyncio.get_event_loop().run_until_complete(run())
def test_consecutive_requests_space_correctly(self): def test_consecutive_requests_space_correctly(self):
async def run():
interval = 0.2 interval = 0.2
async def run():
limiter = RateLimiter(min_interval_seconds=interval) limiter = RateLimiter(min_interval_seconds=interval)
times = [] times = []
for _ in range(5): for _ in range(5):
await limiter.acquire() await limiter.acquire()
times.append(time.monotonic()) times.append(time.monotonic())
return times
times = asyncio.run(run())
for i in range(1, len(times)): for i in range(1, len(times)):
gap = times[i] - times[i - 1] gap = times[i] - times[i - 1]
assert gap >= interval * 0.8 assert gap >= interval * 0.8
asyncio.get_event_loop().run_until_complete(run())
def test_custom_interval(self): def test_custom_interval(self):
async def run(): async def run():
limiter = RateLimiter(min_interval_seconds=0.1) limiter = RateLimiter(min_interval_seconds=0.1)
@ -63,11 +52,9 @@ class TestRateLimiter:
start = time.monotonic() start = time.monotonic()
await limiter.acquire() await limiter.acquire()
elapsed = time.monotonic() - start return time.monotonic() - start
assert elapsed >= 0.05 assert asyncio.run(run()) >= 0.05
asyncio.get_event_loop().run_until_complete(run())
def test_no_delay_after_long_pause(self): def test_no_delay_after_long_pause(self):
async def run(): async def run():
@ -77,8 +64,6 @@ class TestRateLimiter:
start = time.monotonic() start = time.monotonic()
await limiter.acquire() await limiter.acquire()
elapsed = time.monotonic() - start return time.monotonic() - start
assert elapsed < 0.1 assert asyncio.run(run()) < 0.1
asyncio.get_event_loop().run_until_complete(run())

View File

@ -1,4 +1,3 @@
import pytest
class TestSearchResultFormatting: class TestSearchResultFormatting: