DocsMCP - Turn any webpage into LLM-ready context.

Note

This repository is work in progress and can introduce major breaking changes at any time.

This repository uses a uv workspace with independently deployable apps and a shared package:

• apps/backend
• apps/mcp_server
• apps/crawler
• apps/rag
• packages/common

Architecture

+-------------------+          HTTP /api/v1           +----------------------+
| Client / Consumer | -----------------------------> | backend (FastAPI)    |
+-------------------+                                |----------------------|
                                                     | routes               |
                                                     | repositories         |
                                                     | celery producer      |
                                                     | storage service      |
                                                     +----------+-----------+
                                                                |
                         +--------------------------------------+----------------------------------+
                         |                                      |                                  |
                         v                                      v                                  v
               +-------------------+                  +-------------------+              +-------------------+
               | PostgreSQL        |                  | Redis             |              | S3 / Storage      |
               |-------------------|                  |-------------------|              |-------------------|
               | sources           |                  | Celery broker     |              | uploaded files    |
               | pages             |                  | result backend    |              | crawled HTML      |
               | crawl_jobs        |                  +---------+---------+              +---------+---------+
               +-------------------+                            |                                  ^
                                                                |                                  |
                                                                v                                  |
                                                     +----------------------+                      |
                                                     | crawler (Celery)     | ---------------------+
                                                     |----------------------|
                                                     | crawl_source_task    |
                                                     | crawler pipeline     |
                                                     | httpx / Playwright   |
                                                     | shared DB models     |
                                                     +----------+-----------+
                                                                |
                                                                v
                                                     +----------------------+
                                                     | External websites    |
                                                     |----------------------|
                                                     | source URLs crawled  |
                                                     +----------------------+

Build each module independently

uv build --package backend
uv build --package mcp-server
uv build --package crawler

Build artifacts are written to ./dist/.

Run module entrypoints

uv run --package backend backend
uv run --package mcp-server mcp-server
uv run --package crawler crawler

Makefile shortcuts

Command-style:

make backend add httpx redis
make backend add-dev pytest ruff
make backend run
make backend build

make backend run starts the FastAPI dev server with reload:

make backend run BACKEND_HOST=0.0.0.0 BACKEND_PORT=8080

mcp and crawler support the same actions:

make mcp add mcp
make crawler add celery

Variable-style aliases:

make backend-add DEPS="httpx redis"
make mcp-add-dev DEPS="pytest"
make crawler-build

Backend migrations (Alembic)

Alembic is configured in apps/backend/alembic.ini with scripts in apps/backend/migrations.

make backend migrate
make backend revision "create users table"
make backend downgrade
make backend db-check