Jarian Cottingham
645d8820f1
- Wrap RateLimitMiddleware in Middleware() - bare class broke all requests (500) - Clamp/validate num param - ?num=abc no longer returns 500 - Query length cap (500), result counts clamped to 1..20 - Wire lifespan into Starlette so shared httpx client closes on shutdown - Purge stale per-IP rate limit entries to bound memory growth - URL-encode queries in the Playwright scraper path - Lazy lib imports so the package works without optional playwright - Add pyproject.toml (activates ruff/pytest/bandit in CI), README, LICENSE - Remove committed __pycache__, add .gitignore/.dockerignore - Fix SearXNG healthcheck path (/health -> /healthz) and compose docs
135 lines
4.7 KiB
Markdown
135 lines
4.7 KiB
Markdown
# MCP Search Servers
|
|
|
|
Model Context Protocol (MCP) servers that expose **Google** and **DuckDuckGo** search
|
|
as tools for LLM clients (Claude Desktop, Cursor, VS Code, ...), backed by a
|
|
self-hosted [SearXNG](https://github.com/searxng/searxng) metasearch instance.
|
|
|
|
SearXNG aggregates results from Google, DuckDuckGo, Brave, Wikipedia, and more with
|
|
its own request handling, so a single SearXNG container serves both MCP servers —
|
|
no browser fingerprinting, no API keys, fully self-hosted.
|
|
|
|
## Services
|
|
|
|
| Service | Port | MCP tool | Description |
|
|
|-------------------|------|--------------------|--------------------------------------|
|
|
| `google-mcp` | 3001 | `google_search` | Google results via SearXNG |
|
|
| `duckduckgo-mcp` | 3002 | `duckduckgo_search`| DuckDuckGo results via SearXNG |
|
|
| `searxng` | — | — | Internal metasearch engine (not published) |
|
|
|
|
Each server exposes:
|
|
|
|
- `/sse` + `/messages/` — MCP SSE transport
|
|
- `/search?q=...&num=N` — plain HTTP JSON endpoint
|
|
- `/health` — liveness probe
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
cp .env.example .env # fill in SEARXNG_SECRET
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Generate a SearXNG secret:
|
|
|
|
```bash
|
|
python3 -c "import secrets; print(secrets.token_hex(32))"
|
|
```
|
|
|
|
Verify:
|
|
|
|
```bash
|
|
curl http://localhost:3001/health
|
|
curl "http://localhost:3001/search?q=python&num=3"
|
|
```
|
|
|
|
Then point your LLM client at `http://localhost:3001/sse` (Google) and/or
|
|
`http://localhost:3002/sse` (DuckDuckGo). Client configuration examples for
|
|
Claude Desktop, Cursor/Windsurf, and VS Code are in [USAGE.md](USAGE.md).
|
|
|
|
## Tool Contract
|
|
|
|
```json
|
|
{
|
|
"name": "google_search",
|
|
"arguments": { "query": "Python programming language", "num_results": 5 }
|
|
}
|
|
```
|
|
|
|
Returns numbered results with title, URL, and snippet (max 20 results,
|
|
queries capped at 500 characters).
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Description |
|
|
|----------------------|----------------------|------------------------------------------|
|
|
| `SEARXNG_SECRET` | — (required) | SearXNG session/CSRF secret |
|
|
| `RATE_LIMIT_SECONDS` | 5 / 3 | Minimum seconds between search requests |
|
|
| `MCP_PORT` | 3001 / 3002 | Port the MCP server listens on |
|
|
| `SEARXNG_URL` | `http://searxng:8080`| URL of the SearXNG instance |
|
|
|
|
## Security Model
|
|
|
|
- **Internal network** — SearXNG is on a `internal: true` Docker network; only the
|
|
two MCP containers can reach it. It is not published to the host.
|
|
- **Per-IP rate limiting** — HTTP endpoints allow 30 requests/minute per IP
|
|
(`/health` exempt); excess requests get `429`.
|
|
- **Request pacing** — each server also enforces a minimum interval between
|
|
upstream SearXNG queries to avoid throttling.
|
|
- **No auth on `/search` and `/sse`** — the servers assume they are bound to a
|
|
trusted network. Expose them publicly only behind an auth proxy.
|
|
- **Input validation** — query length capped, result counts clamped to 1..20,
|
|
malformed parameters return `400` instead of crashing.
|
|
|
|
## Optional: Playwright Scraping Fallback
|
|
|
|
`lib/google_search.py` and `lib/duckduckgo_search.py` contain a legacy
|
|
Playwright-based scraping path (single shared browser, stealth user agent,
|
|
HTML parsing with multiple fallback strategies). It is **not** used by the
|
|
production servers — Google aggressively blocks headless browsers and SearXNG
|
|
is the working path. If you want to run the scraper directly:
|
|
|
|
```bash
|
|
pip install playwright
|
|
playwright install chromium
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
python3 -m venv .venv && source .venv/bin/activate
|
|
pip install -e ".[dev]"
|
|
|
|
ruff check .
|
|
pytest tests/ -v
|
|
|
|
# End-to-end test (requires running containers)
|
|
python3 integration_test.py
|
|
```
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
├── docker-compose.yml # All 3 services
|
|
├── searxng-settings.yml # SearXNG engine config
|
|
├── google-mcp/
|
|
│ ├── Dockerfile
|
|
│ └── server.py # Google MCP server (SearXNG-backed)
|
|
├── duckduckgo-mcp/
|
|
│ ├── Dockerfile
|
|
│ └── server.py # DuckDuckGo MCP server (SearXNG-backed)
|
|
├── lib/
|
|
│ ├── playwright_manager.py # Single-instance browser manager (optional path)
|
|
│ ├── rate_limiter.py # Async minimum-interval rate limiter
|
|
│ ├── google_search.py # Playwright Google scraper (optional path)
|
|
│ └── duckduckgo_search.py # Playwright DuckDuckGo scraper (optional path)
|
|
├── tests/
|
|
│ ├── test_rate_limiter.py
|
|
│ └── test_search_parsing.py
|
|
├── integration_test.py # End-to-end MCP SSE test
|
|
└── test_client.py # Manual stdio/SSE smoke client
|
|
```
|
|
|
|
## License
|
|
|
|
[MIT](LICENSE)
|