fix: repair middleware wiring, input validation, and resource lifecycle
- Wrap RateLimitMiddleware in Middleware() - bare class broke all requests (500) - Clamp/validate num param - ?num=abc no longer returns 500 - Query length cap (500), result counts clamped to 1..20 - Wire lifespan into Starlette so shared httpx client closes on shutdown - Purge stale per-IP rate limit entries to bound memory growth - URL-encode queries in the Playwright scraper path - Lazy lib imports so the package works without optional playwright - Add pyproject.toml (activates ruff/pytest/bandit in CI), README, LICENSE - Remove committed __pycache__, add .gitignore/.dockerignore - Fix SearXNG healthcheck path (/health -> /healthz) and compose docs
This commit is contained in:
parent
762b2d83db
commit
645d8820f1
11
.dockerignore
Normal file
11
.dockerignore
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
.git
|
||||||
|
__pycache__
|
||||||
|
*.py[cod]
|
||||||
|
.pytest_cache
|
||||||
|
.env
|
||||||
|
tests
|
||||||
|
test_client.py
|
||||||
|
integration_test.py
|
||||||
|
docs
|
||||||
|
*.md
|
||||||
|
docker-compose.yml
|
||||||
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*.egg-info/
|
||||||
|
.pytest_cache/
|
||||||
|
.env
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Jarian Cottingham
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
134
README.md
Normal file
134
README.md
Normal file
@ -0,0 +1,134 @@
|
|||||||
|
# MCP Search Servers
|
||||||
|
|
||||||
|
Model Context Protocol (MCP) servers that expose **Google** and **DuckDuckGo** search
|
||||||
|
as tools for LLM clients (Claude Desktop, Cursor, VS Code, ...), backed by a
|
||||||
|
self-hosted [SearXNG](https://github.com/searxng/searxng) metasearch instance.
|
||||||
|
|
||||||
|
SearXNG aggregates results from Google, DuckDuckGo, Brave, Wikipedia, and more with
|
||||||
|
its own request handling, so a single SearXNG container serves both MCP servers —
|
||||||
|
no browser fingerprinting, no API keys, fully self-hosted.
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
| Service | Port | MCP tool | Description |
|
||||||
|
|-------------------|------|--------------------|--------------------------------------|
|
||||||
|
| `google-mcp` | 3001 | `google_search` | Google results via SearXNG |
|
||||||
|
| `duckduckgo-mcp` | 3002 | `duckduckgo_search`| DuckDuckGo results via SearXNG |
|
||||||
|
| `searxng` | — | — | Internal metasearch engine (not published) |
|
||||||
|
|
||||||
|
Each server exposes:
|
||||||
|
|
||||||
|
- `/sse` + `/messages/` — MCP SSE transport
|
||||||
|
- `/search?q=...&num=N` — plain HTTP JSON endpoint
|
||||||
|
- `/health` — liveness probe
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env # fill in SEARXNG_SECRET
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate a SearXNG secret:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "import secrets; print(secrets.token_hex(32))"
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl http://localhost:3001/health
|
||||||
|
curl "http://localhost:3001/search?q=python&num=3"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then point your LLM client at `http://localhost:3001/sse` (Google) and/or
|
||||||
|
`http://localhost:3002/sse` (DuckDuckGo). Client configuration examples for
|
||||||
|
Claude Desktop, Cursor/Windsurf, and VS Code are in [USAGE.md](USAGE.md).
|
||||||
|
|
||||||
|
## Tool Contract
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"name": "google_search",
|
||||||
|
"arguments": { "query": "Python programming language", "num_results": 5 }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Returns numbered results with title, URL, and snippet (max 20 results,
|
||||||
|
queries capped at 500 characters).
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------------------|----------------------|------------------------------------------|
|
||||||
|
| `SEARXNG_SECRET` | — (required) | SearXNG session/CSRF secret |
|
||||||
|
| `RATE_LIMIT_SECONDS` | 5 / 3 | Minimum seconds between search requests |
|
||||||
|
| `MCP_PORT` | 3001 / 3002 | Port the MCP server listens on |
|
||||||
|
| `SEARXNG_URL` | `http://searxng:8080`| URL of the SearXNG instance |
|
||||||
|
|
||||||
|
## Security Model
|
||||||
|
|
||||||
|
- **Internal network** — SearXNG is on a `internal: true` Docker network; only the
|
||||||
|
two MCP containers can reach it. It is not published to the host.
|
||||||
|
- **Per-IP rate limiting** — HTTP endpoints allow 30 requests/minute per IP
|
||||||
|
(`/health` exempt); excess requests get `429`.
|
||||||
|
- **Request pacing** — each server also enforces a minimum interval between
|
||||||
|
upstream SearXNG queries to avoid throttling.
|
||||||
|
- **No auth on `/search` and `/sse`** — the servers assume they are bound to a
|
||||||
|
trusted network. Expose them publicly only behind an auth proxy.
|
||||||
|
- **Input validation** — query length capped, result counts clamped to 1..20,
|
||||||
|
malformed parameters return `400` instead of crashing.
|
||||||
|
|
||||||
|
## Optional: Playwright Scraping Fallback
|
||||||
|
|
||||||
|
`lib/google_search.py` and `lib/duckduckgo_search.py` contain a legacy
|
||||||
|
Playwright-based scraping path (single shared browser, stealth user agent,
|
||||||
|
HTML parsing with multiple fallback strategies). It is **not** used by the
|
||||||
|
production servers — Google aggressively blocks headless browsers and SearXNG
|
||||||
|
is the working path. If you want to run the scraper directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install playwright
|
||||||
|
playwright install chromium
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv && source .venv/bin/activate
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
|
||||||
|
ruff check .
|
||||||
|
pytest tests/ -v
|
||||||
|
|
||||||
|
# End-to-end test (requires running containers)
|
||||||
|
python3 integration_test.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
├── docker-compose.yml # All 3 services
|
||||||
|
├── searxng-settings.yml # SearXNG engine config
|
||||||
|
├── google-mcp/
|
||||||
|
│ ├── Dockerfile
|
||||||
|
│ └── server.py # Google MCP server (SearXNG-backed)
|
||||||
|
├── duckduckgo-mcp/
|
||||||
|
│ ├── Dockerfile
|
||||||
|
│ └── server.py # DuckDuckGo MCP server (SearXNG-backed)
|
||||||
|
├── lib/
|
||||||
|
│ ├── playwright_manager.py # Single-instance browser manager (optional path)
|
||||||
|
│ ├── rate_limiter.py # Async minimum-interval rate limiter
|
||||||
|
│ ├── google_search.py # Playwright Google scraper (optional path)
|
||||||
|
│ └── duckduckgo_search.py # Playwright DuckDuckGo scraper (optional path)
|
||||||
|
├── tests/
|
||||||
|
│ ├── test_rate_limiter.py
|
||||||
|
│ └── test_search_parsing.py
|
||||||
|
├── integration_test.py # End-to-end MCP SSE test
|
||||||
|
└── test_client.py # Manual stdio/SSE smoke client
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
[MIT](LICENSE)
|
||||||
8
USAGE.md
8
USAGE.md
@ -5,7 +5,7 @@ MCP (Model Context Protocol) servers that expose Google and DuckDuckGo search as
|
|||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd google-mcp
|
# From the repository root
|
||||||
docker compose up -d --build
|
docker compose up -d --build
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -169,9 +169,11 @@ python3 integration_test.py
|
|||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
**No results returned** — SearXNG may be blocked on your network. Check which engines are working:
|
**No results returned** — SearXNG may be blocked on your network. Check which engines are working
|
||||||
|
(SearXNG is on an internal Docker network, so query it from inside its container):
|
||||||
```bash
|
```bash
|
||||||
curl "http://localhost:8080/search?q=test&format=json" | python3 -m json.tool
|
docker compose exec searxng python -c \
|
||||||
|
"import urllib.request; print(urllib.request.urlopen('http://localhost:8080/search?q=test&format=json').read()[:500])"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Services not starting** — Check for port conflicts:
|
**Services not starting** — Check for port conflicts:
|
||||||
|
|||||||
2
conftest.py
Normal file
2
conftest.py
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
# Ensures the repo root is on sys.path so `lib` and the server packages
|
||||||
|
# are importable in tests without installation.
|
||||||
@ -16,7 +16,7 @@ services:
|
|||||||
networks:
|
networks:
|
||||||
- mcp-internal
|
- mcp-internal
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')"]
|
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/healthz')"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
timeout: 10s
|
timeout: 10s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|||||||
@ -1,5 +1,4 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import json
|
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import signal
|
import signal
|
||||||
@ -9,15 +8,15 @@ from contextlib import asynccontextmanager
|
|||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from mcp.server import Server
|
from mcp.server import Server
|
||||||
from mcp.server.sse import SseServerTransport
|
from mcp.server.sse import SseServerTransport
|
||||||
|
from mcp.types import TextContent, Tool
|
||||||
from starlette.applications import Starlette
|
from starlette.applications import Starlette
|
||||||
|
from starlette.middleware import Middleware
|
||||||
from starlette.middleware.base import BaseHTTPMiddleware
|
from starlette.middleware.base import BaseHTTPMiddleware
|
||||||
from starlette.requests import Request
|
from starlette.requests import Request
|
||||||
from starlette.responses import JSONResponse, Response
|
from starlette.responses import JSONResponse, Response
|
||||||
from starlette.routing import Mount, Route
|
from starlette.routing import Mount, Route
|
||||||
from mcp.types import Tool, TextContent
|
|
||||||
|
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
level=logging.INFO,
|
level=logging.INFO,
|
||||||
@ -31,6 +30,19 @@ PORT = int(os.environ.get("MCP_PORT", "3002"))
|
|||||||
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
|
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
|
||||||
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "3"))
|
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "3"))
|
||||||
|
|
||||||
|
DEFAULT_RESULTS = 10
|
||||||
|
MAX_RESULTS = 20
|
||||||
|
MAX_QUERY_LEN = 500
|
||||||
|
|
||||||
|
|
||||||
|
def parse_num_results(value) -> int:
|
||||||
|
"""Parse a result-count value, clamped to 1..MAX_RESULTS."""
|
||||||
|
try:
|
||||||
|
num = int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return DEFAULT_RESULTS
|
||||||
|
return max(1, min(num, MAX_RESULTS))
|
||||||
|
|
||||||
|
|
||||||
# --- HTTPX singleton (#8) ---
|
# --- HTTPX singleton (#8) ---
|
||||||
|
|
||||||
@ -105,10 +117,17 @@ class RateLimitMiddleware(BaseHTTPMiddleware):
|
|||||||
|
|
||||||
client_ip = request.client.host if request.client else "unknown"
|
client_ip = request.client.host if request.client else "unknown"
|
||||||
now = datetime.now(timezone.utc).timestamp()
|
now = datetime.now(timezone.utc).timestamp()
|
||||||
|
cutoff = now - self.window_seconds
|
||||||
|
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
|
if len(self._requests) > 1000:
|
||||||
|
for ip in [
|
||||||
|
ip
|
||||||
|
for ip, ts in self._requests.items()
|
||||||
|
if not ts or ts[-1] <= cutoff
|
||||||
|
]:
|
||||||
|
del self._requests[ip]
|
||||||
timestamps = self._requests[client_ip]
|
timestamps = self._requests[client_ip]
|
||||||
cutoff = now - self.window_seconds
|
|
||||||
self._requests[client_ip] = [t for t in timestamps if t > cutoff]
|
self._requests[client_ip] = [t for t in timestamps if t > cutoff]
|
||||||
if len(self._requests[client_ip]) >= self.max_requests:
|
if len(self._requests[client_ip]) >= self.max_requests:
|
||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
@ -155,15 +174,20 @@ async def do_search(query: str, num_results: int, engine: str):
|
|||||||
# --- HTTP endpoint ---
|
# --- HTTP endpoint ---
|
||||||
|
|
||||||
async def search_http(request: Request):
|
async def search_http(request: Request):
|
||||||
query = request.query_params.get("q", "")
|
query = request.query_params.get("q", "").strip()
|
||||||
if not query:
|
if not query:
|
||||||
return JSONResponse({"error": "'q' parameter required"}, status_code=400)
|
return JSONResponse({"error": "'q' parameter required"}, status_code=400)
|
||||||
|
if len(query) > MAX_QUERY_LEN:
|
||||||
|
return JSONResponse(
|
||||||
|
{"error": f"'q' must be {MAX_QUERY_LEN} characters or fewer"},
|
||||||
|
status_code=400,
|
||||||
|
)
|
||||||
|
|
||||||
num = min(int(request.query_params.get("num", "10")), 20)
|
num = parse_num_results(request.query_params.get("num"))
|
||||||
try:
|
try:
|
||||||
results = await do_search(query, num, "duckduckgo")
|
results = await do_search(query, num, "duckduckgo")
|
||||||
return JSONResponse({"query": query, "results": results})
|
return JSONResponse({"query": query, "results": results})
|
||||||
except Exception as e:
|
except Exception:
|
||||||
logger.exception("Search failed")
|
logger.exception("Search failed")
|
||||||
return JSONResponse({"error": "Search service unavailable"}, status_code=503)
|
return JSONResponse({"error": "Search service unavailable"}, status_code=503)
|
||||||
|
|
||||||
@ -204,14 +228,21 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
|||||||
raise ValueError(f"Unknown tool: {name}")
|
raise ValueError(f"Unknown tool: {name}")
|
||||||
|
|
||||||
query = arguments.get("query", "")
|
query = arguments.get("query", "")
|
||||||
if not query:
|
if not isinstance(query, str) or not query:
|
||||||
return [TextContent(type="text", text="Error: 'query' is required.")]
|
return [TextContent(type="text", text="Error: 'query' is required.")]
|
||||||
|
if len(query) > MAX_QUERY_LEN:
|
||||||
|
return [
|
||||||
|
TextContent(
|
||||||
|
type="text",
|
||||||
|
text=f"Error: query must be {MAX_QUERY_LEN} characters or fewer.",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
num_results = min(int(arguments.get("num_results", 10)), 20)
|
num_results = parse_num_results(arguments.get("num_results"))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
results = await do_search(query, num_results, "duckduckgo")
|
results = await do_search(query, num_results, "duckduckgo")
|
||||||
except Exception as e:
|
except Exception:
|
||||||
logger.exception("Search failed")
|
logger.exception("Search failed")
|
||||||
return [TextContent(type="text", text="Search failed: service unavailable")]
|
return [TextContent(type="text", text="Search failed: service unavailable")]
|
||||||
|
|
||||||
@ -248,13 +279,14 @@ async def handle_sse(request):
|
|||||||
|
|
||||||
starlette_app = Starlette(
|
starlette_app = Starlette(
|
||||||
debug=False,
|
debug=False,
|
||||||
|
lifespan=lifespan,
|
||||||
routes=[
|
routes=[
|
||||||
Route("/health", endpoint=health_check),
|
Route("/health", endpoint=health_check),
|
||||||
Route("/search", endpoint=search_http),
|
Route("/search", endpoint=search_http),
|
||||||
Route("/sse", endpoint=handle_sse),
|
Route("/sse", endpoint=handle_sse),
|
||||||
Mount("/messages/", app=sse.handle_post_message),
|
Mount("/messages/", app=sse.handle_post_message),
|
||||||
],
|
],
|
||||||
middleware=[RateLimitMiddleware],
|
middleware=[Middleware(RateLimitMiddleware)],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -1,5 +1,4 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import json
|
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
import signal
|
import signal
|
||||||
@ -9,15 +8,15 @@ from contextlib import asynccontextmanager
|
|||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from mcp.server import Server
|
from mcp.server import Server
|
||||||
from mcp.server.sse import SseServerTransport
|
from mcp.server.sse import SseServerTransport
|
||||||
|
from mcp.types import TextContent, Tool
|
||||||
from starlette.applications import Starlette
|
from starlette.applications import Starlette
|
||||||
|
from starlette.middleware import Middleware
|
||||||
from starlette.middleware.base import BaseHTTPMiddleware
|
from starlette.middleware.base import BaseHTTPMiddleware
|
||||||
from starlette.requests import Request
|
from starlette.requests import Request
|
||||||
from starlette.responses import JSONResponse, Response
|
from starlette.responses import JSONResponse, Response
|
||||||
from starlette.routing import Mount, Route
|
from starlette.routing import Mount, Route
|
||||||
from mcp.types import Tool, TextContent
|
|
||||||
|
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
level=logging.INFO,
|
level=logging.INFO,
|
||||||
@ -31,6 +30,19 @@ PORT = int(os.environ.get("MCP_PORT", "3001"))
|
|||||||
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
|
SEARXNG_URL = os.environ.get("SEARXNG_URL", "http://searxng:8080")
|
||||||
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "5"))
|
RATE_LIMIT_SECONDS = float(os.environ.get("RATE_LIMIT_SECONDS", "5"))
|
||||||
|
|
||||||
|
DEFAULT_RESULTS = 10
|
||||||
|
MAX_RESULTS = 20
|
||||||
|
MAX_QUERY_LEN = 500
|
||||||
|
|
||||||
|
|
||||||
|
def parse_num_results(value) -> int:
|
||||||
|
"""Parse a result-count value, clamped to 1..MAX_RESULTS."""
|
||||||
|
try:
|
||||||
|
num = int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return DEFAULT_RESULTS
|
||||||
|
return max(1, min(num, MAX_RESULTS))
|
||||||
|
|
||||||
|
|
||||||
# --- HTTPX singleton (#8) ---
|
# --- HTTPX singleton (#8) ---
|
||||||
|
|
||||||
@ -105,10 +117,17 @@ class RateLimitMiddleware(BaseHTTPMiddleware):
|
|||||||
|
|
||||||
client_ip = request.client.host if request.client else "unknown"
|
client_ip = request.client.host if request.client else "unknown"
|
||||||
now = datetime.now(timezone.utc).timestamp()
|
now = datetime.now(timezone.utc).timestamp()
|
||||||
|
cutoff = now - self.window_seconds
|
||||||
|
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
|
if len(self._requests) > 1000:
|
||||||
|
for ip in [
|
||||||
|
ip
|
||||||
|
for ip, ts in self._requests.items()
|
||||||
|
if not ts or ts[-1] <= cutoff
|
||||||
|
]:
|
||||||
|
del self._requests[ip]
|
||||||
timestamps = self._requests[client_ip]
|
timestamps = self._requests[client_ip]
|
||||||
cutoff = now - self.window_seconds
|
|
||||||
self._requests[client_ip] = [t for t in timestamps if t > cutoff]
|
self._requests[client_ip] = [t for t in timestamps if t > cutoff]
|
||||||
if len(self._requests[client_ip]) >= self.max_requests:
|
if len(self._requests[client_ip]) >= self.max_requests:
|
||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
@ -155,15 +174,20 @@ async def do_search(query: str, num_results: int, engine: str):
|
|||||||
# --- HTTP endpoint ---
|
# --- HTTP endpoint ---
|
||||||
|
|
||||||
async def search_http(request: Request):
|
async def search_http(request: Request):
|
||||||
query = request.query_params.get("q", "")
|
query = request.query_params.get("q", "").strip()
|
||||||
if not query:
|
if not query:
|
||||||
return JSONResponse({"error": "'q' parameter required"}, status_code=400)
|
return JSONResponse({"error": "'q' parameter required"}, status_code=400)
|
||||||
|
if len(query) > MAX_QUERY_LEN:
|
||||||
|
return JSONResponse(
|
||||||
|
{"error": f"'q' must be {MAX_QUERY_LEN} characters or fewer"},
|
||||||
|
status_code=400,
|
||||||
|
)
|
||||||
|
|
||||||
num = min(int(request.query_params.get("num", "10")), 20)
|
num = parse_num_results(request.query_params.get("num"))
|
||||||
try:
|
try:
|
||||||
results = await do_search(query, num, "google")
|
results = await do_search(query, num, "google")
|
||||||
return JSONResponse({"query": query, "results": results})
|
return JSONResponse({"query": query, "results": results})
|
||||||
except Exception as e:
|
except Exception:
|
||||||
logger.exception("Search failed")
|
logger.exception("Search failed")
|
||||||
return JSONResponse({"error": "Search service unavailable"}, status_code=503)
|
return JSONResponse({"error": "Search service unavailable"}, status_code=503)
|
||||||
|
|
||||||
@ -204,14 +228,21 @@ async def call_tool(name: str, arguments: dict) -> list[TextContent]:
|
|||||||
raise ValueError(f"Unknown tool: {name}")
|
raise ValueError(f"Unknown tool: {name}")
|
||||||
|
|
||||||
query = arguments.get("query", "")
|
query = arguments.get("query", "")
|
||||||
if not query:
|
if not isinstance(query, str) or not query:
|
||||||
return [TextContent(type="text", text="Error: 'query' is required.")]
|
return [TextContent(type="text", text="Error: 'query' is required.")]
|
||||||
|
if len(query) > MAX_QUERY_LEN:
|
||||||
|
return [
|
||||||
|
TextContent(
|
||||||
|
type="text",
|
||||||
|
text=f"Error: query must be {MAX_QUERY_LEN} characters or fewer.",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
num_results = min(int(arguments.get("num_results", 10)), 20)
|
num_results = parse_num_results(arguments.get("num_results"))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
results = await do_search(query, num_results, "google")
|
results = await do_search(query, num_results, "google")
|
||||||
except Exception as e:
|
except Exception:
|
||||||
logger.exception("Search failed")
|
logger.exception("Search failed")
|
||||||
return [TextContent(type="text", text="Search failed: service unavailable")]
|
return [TextContent(type="text", text="Search failed: service unavailable")]
|
||||||
|
|
||||||
@ -248,13 +279,14 @@ async def handle_sse(request):
|
|||||||
|
|
||||||
starlette_app = Starlette(
|
starlette_app = Starlette(
|
||||||
debug=False,
|
debug=False,
|
||||||
|
lifespan=lifespan,
|
||||||
routes=[
|
routes=[
|
||||||
Route("/health", endpoint=health_check),
|
Route("/health", endpoint=health_check),
|
||||||
Route("/search", endpoint=search_http),
|
Route("/search", endpoint=search_http),
|
||||||
Route("/sse", endpoint=handle_sse),
|
Route("/sse", endpoint=handle_sse),
|
||||||
Mount("/messages/", app=sse.handle_post_message),
|
Mount("/messages/", app=sse.handle_post_message),
|
||||||
],
|
],
|
||||||
middleware=[RateLimitMiddleware],
|
middleware=[Middleware(RateLimitMiddleware)],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -3,6 +3,7 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
import aiohttp
|
import aiohttp
|
||||||
|
|
||||||
|
|
||||||
@ -76,7 +77,7 @@ async def test_server(name, host, port, tool_name, query):
|
|||||||
"jsonrpc": "2.0", "id": 1, "method": "initialize",
|
"jsonrpc": "2.0", "id": 1, "method": "initialize",
|
||||||
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
|
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
|
||||||
"clientInfo": {"name": "test", "version": "1.0"}},
|
"clientInfo": {"name": "test", "version": "1.0"}},
|
||||||
}) as resp:
|
}):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
init = await collector.wait_for_id(1, timeout=10)
|
init = await collector.wait_for_id(1, timeout=10)
|
||||||
@ -88,25 +89,31 @@ async def test_server(name, host, port, tool_name, query):
|
|||||||
# Initialized notification
|
# Initialized notification
|
||||||
async with session.post(messages_endpoint, json={
|
async with session.post(messages_endpoint, json={
|
||||||
"jsonrpc": "2.0", "method": "notifications/initialized",
|
"jsonrpc": "2.0", "method": "notifications/initialized",
|
||||||
}) as resp:
|
}):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
# List tools
|
# List tools
|
||||||
async with session.post(messages_endpoint, json={
|
async with session.post(messages_endpoint, json={
|
||||||
"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {},
|
"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {},
|
||||||
}) as resp:
|
}):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
tools = await collector.wait_for_id(2, timeout=10)
|
tools = await collector.wait_for_id(2, timeout=10)
|
||||||
if tools:
|
if tools:
|
||||||
print(f" Tools: {[t['name'] for t in tools.get('result', {}).get('tools', [])]}")
|
tool_names = [
|
||||||
|
t["name"] for t in tools.get("result", {}).get("tools", [])
|
||||||
|
]
|
||||||
|
print(f" Tools: {tool_names}")
|
||||||
|
|
||||||
# Call search
|
# Call search
|
||||||
print(f" Searching: '{query}'")
|
print(f" Searching: '{query}'")
|
||||||
async with session.post(messages_endpoint, json={
|
async with session.post(messages_endpoint, json={
|
||||||
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
|
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
|
||||||
"params": {"name": tool_name, "arguments": {"query": query, "num_results": 5}},
|
"params": {
|
||||||
}) as resp:
|
"name": tool_name,
|
||||||
|
"arguments": {"query": query, "num_results": 5},
|
||||||
|
},
|
||||||
|
}):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
result = await collector.wait_for_id(3, timeout=30)
|
result = await collector.wait_for_id(3, timeout=30)
|
||||||
@ -145,11 +152,14 @@ async def main():
|
|||||||
print("Waiting for services to start...")
|
print("Waiting for services to start...")
|
||||||
await asyncio.sleep(5)
|
await asyncio.sleep(5)
|
||||||
|
|
||||||
results["google"] = await test_server("Google", host, 3001, "google_search", "Python programming language")
|
query = "Python programming language"
|
||||||
|
results["google"] = await test_server("Google", host, 3001, "google_search", query)
|
||||||
|
|
||||||
await asyncio.sleep(2)
|
await asyncio.sleep(2)
|
||||||
|
|
||||||
results["duckduckgo"] = await test_server("DuckDuckGo", host, 3002, "duckduckgo_search", "Python programming language")
|
results["duckduckgo"] = await test_server(
|
||||||
|
"DuckDuckGo", host, 3002, "duckduckgo_search", query
|
||||||
|
)
|
||||||
|
|
||||||
print(f"\n{'='*60}")
|
print(f"\n{'='*60}")
|
||||||
print("SUMMARY")
|
print("SUMMARY")
|
||||||
|
|||||||
@ -1,11 +1,25 @@
|
|||||||
from lib.playwright_manager import PlaywrightManager
|
"""Shared search utilities.
|
||||||
from lib.rate_limiter import RateLimiter
|
|
||||||
from lib.google_search import GoogleSearch
|
|
||||||
from lib.duckduckgo_search import DuckDuckGoSearch
|
|
||||||
|
|
||||||
__all__ = [
|
Playwright-based scrapers are imported lazily so the package (and its
|
||||||
"PlaywrightManager",
|
rate limiter) works without the optional ``playwright`` dependency.
|
||||||
"RateLimiter",
|
"""
|
||||||
"GoogleSearch",
|
|
||||||
"DuckDuckGoSearch",
|
from lib.rate_limiter import RateLimiter
|
||||||
]
|
|
||||||
|
__all__ = ["PlaywrightManager", "RateLimiter", "GoogleSearch", "DuckDuckGoSearch"]
|
||||||
|
|
||||||
|
|
||||||
|
def __getattr__(name):
|
||||||
|
if name == "PlaywrightManager":
|
||||||
|
from lib.playwright_manager import PlaywrightManager
|
||||||
|
|
||||||
|
return PlaywrightManager
|
||||||
|
if name == "GoogleSearch":
|
||||||
|
from lib.google_search import GoogleSearch
|
||||||
|
|
||||||
|
return GoogleSearch
|
||||||
|
if name == "DuckDuckGoSearch":
|
||||||
|
from lib.duckduckgo_search import DuckDuckGoSearch
|
||||||
|
|
||||||
|
return DuckDuckGoSearch
|
||||||
|
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,6 +1,6 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
from typing import List, Dict, Any, Optional
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
from lib.playwright_manager import PlaywrightManager
|
from lib.playwright_manager import PlaywrightManager
|
||||||
from lib.rate_limiter import RateLimiter
|
from lib.rate_limiter import RateLimiter
|
||||||
|
|||||||
@ -1,7 +1,8 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
from typing import List, Dict, Any
|
import urllib.parse
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
from lib.playwright_manager import PlaywrightManager
|
from lib.playwright_manager import PlaywrightManager
|
||||||
from lib.rate_limiter import RateLimiter
|
from lib.rate_limiter import RateLimiter
|
||||||
@ -29,7 +30,10 @@ class GoogleSearch:
|
|||||||
try:
|
try:
|
||||||
page = await PlaywrightManager.get_page(stealth=True)
|
page = await PlaywrightManager.get_page(stealth=True)
|
||||||
|
|
||||||
url_params = f"{GOOGLE_URL}?q={query}&num={min(num_results, 20)}&hl={language}"
|
url_params = (
|
||||||
|
f"{GOOGLE_URL}?q={urllib.parse.quote(query)}"
|
||||||
|
f"&num={min(num_results, 20)}&hl={language}"
|
||||||
|
)
|
||||||
logger.info(f"Navigating to Google search: {query}")
|
logger.info(f"Navigating to Google search: {query}")
|
||||||
|
|
||||||
await page.goto(url_params, wait_until="domcontentloaded", timeout=30000)
|
await page.goto(url_params, wait_until="domcontentloaded", timeout=30000)
|
||||||
@ -98,8 +102,6 @@ class GoogleSearch:
|
|||||||
|
|
||||||
def _clean_url(self, raw_url: str) -> str:
|
def _clean_url(self, raw_url: str) -> str:
|
||||||
"""Decode Google redirect URL."""
|
"""Decode Google redirect URL."""
|
||||||
import urllib.parse
|
|
||||||
|
|
||||||
raw_url = urllib.parse.unquote(raw_url)
|
raw_url = urllib.parse.unquote(raw_url)
|
||||||
# Remove tracking parameters
|
# Remove tracking parameters
|
||||||
url = re.sub(r"&[a-z_]+=.*$", "", raw_url)
|
url = re.sub(r"&[a-z_]+=.*$", "", raw_url)
|
||||||
@ -118,7 +120,13 @@ class GoogleSearch:
|
|||||||
for i, result in enumerate(results):
|
for i, result in enumerate(results):
|
||||||
for block in snippet_blocks:
|
for block in snippet_blocks:
|
||||||
clean = re.sub(r'<[^>]+>', "", block).strip()
|
clean = re.sub(r'<[^>]+>', "", block).strip()
|
||||||
if 50 < len(clean) < 300 and result["url"] in html[max(0, html.find(clean) - 500):html.find(clean) + 500]:
|
if not 50 < len(clean) < 300:
|
||||||
|
continue
|
||||||
|
pos = html.find(clean)
|
||||||
|
if pos == -1:
|
||||||
|
continue
|
||||||
|
window = html[max(0, pos - 500): pos + 500]
|
||||||
|
if result["url"] in window:
|
||||||
result["snippet"] = clean[:200]
|
result["snippet"] = clean[:200]
|
||||||
break
|
break
|
||||||
return results
|
return results
|
||||||
|
|||||||
@ -2,7 +2,7 @@ import asyncio
|
|||||||
import logging
|
import logging
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
from playwright.async_api import async_playwright, Browser, BrowserContext, Page
|
from playwright.async_api import Browser, BrowserContext, Page, async_playwright
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@ -75,7 +75,7 @@ class PlaywrightManager:
|
|||||||
stealth = Stealth()
|
stealth = Stealth()
|
||||||
await stealth.apply_stealth_async(page)
|
await stealth.apply_stealth_async(page)
|
||||||
logger.debug("Stealth mode applied to page")
|
logger.debug("Stealth mode applied to page")
|
||||||
except (ImportError, Exception) as e:
|
except Exception as e:
|
||||||
logger.warning(f"Could not apply stealth: {e}")
|
logger.warning(f"Could not apply stealth: {e}")
|
||||||
return page
|
return page
|
||||||
|
|
||||||
|
|||||||
37
pyproject.toml
Normal file
37
pyproject.toml
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
[build-system]
|
||||||
|
requires = ["setuptools>=68"]
|
||||||
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
|
[project]
|
||||||
|
name = "mcp-search-servers"
|
||||||
|
version = "1.1.0"
|
||||||
|
description = "MCP servers exposing Google and DuckDuckGo search via SearXNG"
|
||||||
|
readme = "README.md"
|
||||||
|
license = { text = "MIT" }
|
||||||
|
requires-python = ">=3.10"
|
||||||
|
dependencies = [
|
||||||
|
"mcp>=1.0.0,<2.0.0",
|
||||||
|
"httpx>=0.27.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = [
|
||||||
|
"pytest>=8.0",
|
||||||
|
"ruff>=0.4",
|
||||||
|
]
|
||||||
|
playwright = [
|
||||||
|
"playwright>=1.44",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.setuptools]
|
||||||
|
packages = ["lib"]
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
line-length = 100
|
||||||
|
target-version = "py310"
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "W", "I"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
@ -7,10 +7,8 @@ Usage:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import time
|
|
||||||
|
|
||||||
|
|
||||||
async def test_mcp_server(service_name: str):
|
async def test_mcp_server(service_name: str):
|
||||||
@ -36,7 +34,13 @@ async def test():
|
|||||||
await asyncio.sleep(0.5)
|
await asyncio.sleep(0.5)
|
||||||
init_req = json.loads(await read.__anext__())
|
init_req = json.loads(await read.__anext__())
|
||||||
# Send initialize response
|
# Send initialize response
|
||||||
init_result = {{'jsonrpc': '2.0', 'id': init_req['id'], 'result': {{'protocolVersion': '2024-11-05', 'capabilities': {{}}, 'serverInfo': {{'name': 'test', 'version': '1.0'}}}}}}
|
init_result = {{
|
||||||
|
'jsonrpc': '2.0', 'id': init_req['id'],
|
||||||
|
'result': {{
|
||||||
|
'protocolVersion': '2024-11-05', 'capabilities': {{}},
|
||||||
|
'serverInfo': {{'name': 'test', 'version': '1.0'}},
|
||||||
|
}},
|
||||||
|
}}
|
||||||
await write.send(json.dumps(init_result))
|
await write.send(json.dumps(init_result))
|
||||||
# Client sends initialized notification
|
# Client sends initialized notification
|
||||||
await read.__anext__()
|
await read.__anext__()
|
||||||
@ -68,7 +72,6 @@ async def test():
|
|||||||
async def test_with_curl(service_name: str):
|
async def test_with_curl(service_name: str):
|
||||||
"""Test using docker exec to run a Python test directly in the container."""
|
"""Test using docker exec to run a Python test directly in the container."""
|
||||||
container_name = f"{service_name}-mcp"
|
container_name = f"{service_name}-mcp"
|
||||||
module = service_name.replace("-", "_")
|
|
||||||
|
|
||||||
test_script = f"""
|
test_script = f"""
|
||||||
import sys, asyncio, os
|
import sys, asyncio, os
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,19 +1,9 @@
|
|||||||
import asyncio
|
import asyncio
|
||||||
import time
|
import time
|
||||||
|
|
||||||
import pytest
|
|
||||||
import pytest_asyncio
|
|
||||||
|
|
||||||
from lib.rate_limiter import RateLimiter
|
from lib.rate_limiter import RateLimiter
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def event_loop():
|
|
||||||
loop = asyncio.new_event_loop()
|
|
||||||
yield loop
|
|
||||||
loop.close()
|
|
||||||
|
|
||||||
|
|
||||||
class TestRateLimiter:
|
class TestRateLimiter:
|
||||||
"""Test rate limiter enforces minimum delays between requests."""
|
"""Test rate limiter enforces minimum delays between requests."""
|
||||||
|
|
||||||
@ -23,9 +13,9 @@ class TestRateLimiter:
|
|||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
await limiter.acquire()
|
await limiter.acquire()
|
||||||
elapsed = time.monotonic() - start
|
elapsed = time.monotonic() - start
|
||||||
assert elapsed < 0.1
|
return elapsed
|
||||||
|
|
||||||
asyncio.get_event_loop().run_until_complete(run())
|
assert asyncio.run(run()) < 0.1
|
||||||
|
|
||||||
def test_enforces_minimum_interval(self):
|
def test_enforces_minimum_interval(self):
|
||||||
async def run():
|
async def run():
|
||||||
@ -34,28 +24,27 @@ class TestRateLimiter:
|
|||||||
|
|
||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
await limiter.acquire()
|
await limiter.acquire()
|
||||||
elapsed = time.monotonic() - start
|
return time.monotonic() - start
|
||||||
|
|
||||||
assert elapsed >= 0.25
|
assert asyncio.run(run()) >= 0.25
|
||||||
|
|
||||||
asyncio.get_event_loop().run_until_complete(run())
|
|
||||||
|
|
||||||
def test_consecutive_requests_space_correctly(self):
|
def test_consecutive_requests_space_correctly(self):
|
||||||
async def run():
|
|
||||||
interval = 0.2
|
interval = 0.2
|
||||||
|
|
||||||
|
async def run():
|
||||||
limiter = RateLimiter(min_interval_seconds=interval)
|
limiter = RateLimiter(min_interval_seconds=interval)
|
||||||
|
|
||||||
times = []
|
times = []
|
||||||
for _ in range(5):
|
for _ in range(5):
|
||||||
await limiter.acquire()
|
await limiter.acquire()
|
||||||
times.append(time.monotonic())
|
times.append(time.monotonic())
|
||||||
|
return times
|
||||||
|
|
||||||
|
times = asyncio.run(run())
|
||||||
for i in range(1, len(times)):
|
for i in range(1, len(times)):
|
||||||
gap = times[i] - times[i - 1]
|
gap = times[i] - times[i - 1]
|
||||||
assert gap >= interval * 0.8
|
assert gap >= interval * 0.8
|
||||||
|
|
||||||
asyncio.get_event_loop().run_until_complete(run())
|
|
||||||
|
|
||||||
def test_custom_interval(self):
|
def test_custom_interval(self):
|
||||||
async def run():
|
async def run():
|
||||||
limiter = RateLimiter(min_interval_seconds=0.1)
|
limiter = RateLimiter(min_interval_seconds=0.1)
|
||||||
@ -63,11 +52,9 @@ class TestRateLimiter:
|
|||||||
|
|
||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
await limiter.acquire()
|
await limiter.acquire()
|
||||||
elapsed = time.monotonic() - start
|
return time.monotonic() - start
|
||||||
|
|
||||||
assert elapsed >= 0.05
|
assert asyncio.run(run()) >= 0.05
|
||||||
|
|
||||||
asyncio.get_event_loop().run_until_complete(run())
|
|
||||||
|
|
||||||
def test_no_delay_after_long_pause(self):
|
def test_no_delay_after_long_pause(self):
|
||||||
async def run():
|
async def run():
|
||||||
@ -77,8 +64,6 @@ class TestRateLimiter:
|
|||||||
|
|
||||||
start = time.monotonic()
|
start = time.monotonic()
|
||||||
await limiter.acquire()
|
await limiter.acquire()
|
||||||
elapsed = time.monotonic() - start
|
return time.monotonic() - start
|
||||||
|
|
||||||
assert elapsed < 0.1
|
assert asyncio.run(run()) < 0.1
|
||||||
|
|
||||||
asyncio.get_event_loop().run_until_complete(run())
|
|
||||||
|
|||||||
@ -1,4 +1,3 @@
|
|||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
class TestSearchResultFormatting:
|
class TestSearchResultFormatting:
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user