Jarian Cottingham 645d8820f1 fix: repair middleware wiring, input validation, and resource lifecycle

- Wrap RateLimitMiddleware in Middleware() - bare class broke all requests (500)
- Clamp/validate num param - ?num=abc no longer returns 500
- Query length cap (500), result counts clamped to 1..20
- Wire lifespan into Starlette so shared httpx client closes on shutdown
- Purge stale per-IP rate limit entries to bound memory growth
- URL-encode queries in the Playwright scraper path
- Lazy lib imports so the package works without optional playwright
- Add pyproject.toml (activates ruff/pytest/bandit in CI), README, LICENSE
- Remove committed __pycache__, add .gitignore/.dockerignore
- Fix SearXNG healthcheck path (/health -> /healthz) and compose docs

2026-08-20 23:27:27 +00:00

4.7 KiB

Raw Permalink Blame History

MCP Search Servers

Model Context Protocol (MCP) servers that expose Google and DuckDuckGo search as tools for LLM clients (Claude Desktop, Cursor, VS Code, ...), backed by a self-hosted SearXNG metasearch instance.

SearXNG aggregates results from Google, DuckDuckGo, Brave, Wikipedia, and more with its own request handling, so a single SearXNG container serves both MCP servers — no browser fingerprinting, no API keys, fully self-hosted.

Services

Service	Port	MCP tool	Description
`google-mcp`	3001	`google_search`	Google results via SearXNG
`duckduckgo-mcp`	3002	`duckduckgo_search`	DuckDuckGo results via SearXNG
`searxng`	—	—	Internal metasearch engine (not published)

Each server exposes:

/sse + /messages/ — MCP SSE transport
/search?q=...&num=N — plain HTTP JSON endpoint
/health — liveness probe

Quick Start

cp .env.example .env        # fill in SEARXNG_SECRET
docker compose up -d --build

Generate a SearXNG secret:

python3 -c "import secrets; print(secrets.token_hex(32))"

Verify:

curl http://localhost:3001/health
curl "http://localhost:3001/search?q=python&num=3"

Then point your LLM client at http://localhost:3001/sse (Google) and/or http://localhost:3002/sse (DuckDuckGo). Client configuration examples for Claude Desktop, Cursor/Windsurf, and VS Code are in USAGE.md.

Tool Contract

{
  "name": "google_search",
  "arguments": { "query": "Python programming language", "num_results": 5 }
}

Returns numbered results with title, URL, and snippet (max 20 results, queries capped at 500 characters).

Configuration

Variable	Default	Description
`SEARXNG_SECRET`	— (required)	SearXNG session/CSRF secret
`RATE_LIMIT_SECONDS`	5 / 3	Minimum seconds between search requests
`MCP_PORT`	3001 / 3002	Port the MCP server listens on
`SEARXNG_URL`	`http://searxng:8080`	URL of the SearXNG instance

Security Model

Internal network — SearXNG is on a internal: true Docker network; only the two MCP containers can reach it. It is not published to the host.
Per-IP rate limiting — HTTP endpoints allow 30 requests/minute per IP (/health exempt); excess requests get 429.
Request pacing — each server also enforces a minimum interval between upstream SearXNG queries to avoid throttling.
No auth on /search and /sse — the servers assume they are bound to a trusted network. Expose them publicly only behind an auth proxy.
Input validation — query length capped, result counts clamped to 1..20, malformed parameters return 400 instead of crashing.

Optional: Playwright Scraping Fallback

lib/google_search.py and lib/duckduckgo_search.py contain a legacy Playwright-based scraping path (single shared browser, stealth user agent, HTML parsing with multiple fallback strategies). It is not used by the production servers — Google aggressively blocks headless browsers and SearXNG is the working path. If you want to run the scraper directly:

pip install playwright
playwright install chromium

Development

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

ruff check .
pytest tests/ -v

# End-to-end test (requires running containers)
python3 integration_test.py

Project Structure

├── docker-compose.yml        # All 3 services
├── searxng-settings.yml      # SearXNG engine config
├── google-mcp/
│   ├── Dockerfile
│   └── server.py             # Google MCP server (SearXNG-backed)
├── duckduckgo-mcp/
│   ├── Dockerfile
│   └── server.py             # DuckDuckGo MCP server (SearXNG-backed)
├── lib/
│   ├── playwright_manager.py # Single-instance browser manager (optional path)
│   ├── rate_limiter.py       # Async minimum-interval rate limiter
│   ├── google_search.py      # Playwright Google scraper (optional path)
│   └── duckduckgo_search.py  # Playwright DuckDuckGo scraper (optional path)
├── tests/
│   ├── test_rate_limiter.py
│   └── test_search_parsing.py
├── integration_test.py       # End-to-end MCP SSE test
└── test_client.py            # Manual stdio/SSE smoke client

License

MIT

4.7 KiB Raw Permalink Blame History