Skip to content

API.md

Latest commit

 

History

History
1034 lines (782 loc) · 19.2 KB

API.md

File metadata and controls

1034 lines (782 loc) · 19.2 KB

HFDownloader REST API Reference

Complete REST API documentation for hfdownloader serve.

Table of Contents

Overview

Start the server:

hfdownloader serve --port 8080

The server provides:

  • REST API for download management
  • WebSocket for real-time progress updates
  • Web UI at the root URL

Content Types

  • Request bodies: application/json
  • Response bodies: application/json
  • WebSocket messages: JSON

Authentication

Basic Authentication (Optional)

Enable with server flags:

hfdownloader serve --auth-user admin --auth-pass secret123

All requests require the Authorization header:

Authorization: Basic YWRtaW46c2VjcmV0MTIz

HuggingFace Token

For private/gated models, provide via:

  1. Server flag: hfdownloader serve -t hf_xxxxx
  2. Settings API: POST /api/settings

Base URL

http://localhost:8080/api

Endpoints

Health Check

Check server status.

GET /api/health

Response 200 OK

{
  "status": "ok",
  "version": "3.0.0",
  "time": "2024-01-15T10:30:00Z"
}

Example

curl http://localhost:8080/api/health

Downloads

POST /api/download

Start a new download job.

Request Body

Field Type Required Default Description
repo string Yes Repository ID (owner/name)
revision string No main Branch, tag, or commit
dataset boolean No false Treat as dataset
filters string[] No [] File filter patterns
excludes string[] No [] Exclude patterns
appendFilterSubdir boolean No false Create filter subdirs
dryRun boolean No false Plan only

Filter Syntax

Filters can be embedded in repo name:

{ "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF:q4_k_m,q5_k_m" }

Or as separate field:

{
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "filters": ["q4_k_m", "q5_k_m"]
}

Response 202 Accepted (New job)

{
  "id": "a1b2c3d4e5f6",
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "revision": "main",
  "isDataset": false,
  "filters": ["q4_k_m"],
  "excludes": [],
  "outputDir": "/home/user/.cache/huggingface/hub",
  "status": "queued",
  "progress": {
    "totalFiles": 0,
    "completedFiles": 0,
    "totalBytes": 0,
    "downloadedBytes": 0,
    "bytesPerSecond": 0
  },
  "error": "",
  "createdAt": "2024-01-15T10:30:00Z",
  "startedAt": null,
  "endedAt": null,
  "files": []
}

Response 200 OK (Existing job)

{
  "job": { /* job object */ },
  "message": "Download already in progress"
}

Examples

# Basic download
curl -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{"repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF"}'

# With filters
curl -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{
    "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
    "filters": ["q4_k_m", "q5_k_m"]
  }'

# Dataset download
curl -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{"repo": "facebook/flores", "dataset": true}'

# Specific revision
curl -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{"repo": "CompVis/stable-diffusion-v1-4", "revision": "fp16"}'

POST /api/plan

Get download plan without starting download.

Request Body

Same as /api/download.

Response 200 OK

{
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "revision": "main",
  "files": [
    {
      "path": "config.json",
      "size": 1024,
      "lfs": false
    },
    {
      "path": "mistral-7b.Q4_K_M.gguf",
      "size": 4368438272,
      "lfs": true
    }
  ],
  "totalSize": 4368439296,
  "totalFiles": 2
}

Example

curl -X POST http://localhost:8080/api/plan \
  -H "Content-Type: application/json" \
  -d '{"repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "filters": ["q4_k_m"]}'

Jobs

GET /api/jobs

List all download jobs.

Response 200 OK

{
  "jobs": [
    {
      "id": "a1b2c3d4e5f6",
      "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
      "revision": "main",
      "isDataset": false,
      "filters": ["q4_k_m"],
      "excludes": [],
      "outputDir": "/home/user/.cache/huggingface/hub",
      "status": "running",
      "progress": {
        "totalFiles": 3,
        "completedFiles": 1,
        "totalBytes": 4500000000,
        "downloadedBytes": 1500000000,
        "bytesPerSecond": 50000000
      },
      "error": "",
      "createdAt": "2024-01-15T10:30:00Z",
      "startedAt": "2024-01-15T10:30:01Z",
      "endedAt": null,
      "files": [
        {
          "path": "config.json",
          "totalBytes": 1024,
          "downloaded": 1024,
          "status": "complete"
        },
        {
          "path": "mistral-7b.Q4_K_M.gguf",
          "totalBytes": 4368438272,
          "downloaded": 1500000000,
          "status": "downloading"
        }
      ]
    }
  ],
  "count": 1
}

Example

curl http://localhost:8080/api/jobs

GET /api/jobs/{id}

Get specific job details.

Path Parameters

Parameter Type Description
id string Job ID

Response 200 OK

{
  "id": "a1b2c3d4e5f6",
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "status": "running",
  /* ... full job object ... */
}

Response 404 Not Found

{
  "error": "Job not found"
}

Example

curl http://localhost:8080/api/jobs/a1b2c3d4e5f6

DELETE /api/jobs/{id}

Cancel a running or queued job.

Response 200 OK

{
  "success": true,
  "message": "Job cancelled"
}

Response 404 Not Found

{
  "error": "Job not found or already completed"
}

Example

curl -X DELETE http://localhost:8080/api/jobs/a1b2c3d4e5f6

POST /api/jobs/{id}/pause

Pause a running job.

Response 200 OK

{
  "success": true,
  "message": "Job paused"
}

Response 404 Not Found

{
  "error": "Job not found or not running"
}

Example

curl -X POST http://localhost:8080/api/jobs/a1b2c3d4e5f6/pause

POST /api/jobs/{id}/resume

Resume a paused job.

Response 200 OK

{
  "success": true,
  "message": "Job resumed"
}

Response 404 Not Found

{
  "error": "Job not found or not paused"
}

Note: Resumed jobs restart from queued status. Progress is reset, but already-downloaded files are automatically skipped.

Example

curl -X POST http://localhost:8080/api/jobs/a1b2c3d4e5f6/resume

Job Status Lifecycle

queued ─────► running ─────► completed
                │
                ├─────► failed
                │
                ├─────► cancelled
                │
                └─────► paused ─────► queued ─► running ─► ...
Status Description
queued Waiting to start
running Download in progress
paused Paused by user
completed Finished successfully
failed Error occurred
cancelled Cancelled by user

Settings

GET /api/settings

Get current server settings.

Response 200 OK

{
  "token": "********mnop",
  "cacheDir": "/home/user/.cache/huggingface/hub",
  "connections": 8,
  "maxActive": 3,
  "multipartThreshold": "32MiB",
  "verify": "size",
  "retries": 4,
  "endpoint": ""
}

Note: Token is masked for security (shows ******** + last 4 characters).

Example

curl http://localhost:8080/api/settings

POST /api/settings

Update server settings.

Request Body

Field Type Description
token string HuggingFace token
connections integer Connections per file
maxActive integer Max concurrent downloads
multipartThreshold string Min size for multipart
verify string Verification: none, size, sha256
retries integer Retry attempts

Security Restrictions

  • cacheDir cannot be changed via API
  • modelsDir cannot be changed via API
  • datasetsDir cannot be changed via API

Response 200 OK

{
  "success": true,
  "message": "Settings updated"
}

Example

curl -X POST http://localhost:8080/api/settings \
  -H "Content-Type: application/json" \
  -d '{
    "token": "hf_xxxxx",
    "connections": 16,
    "maxActive": 8
  }'

Analyzer

GET /api/analyze/{repo}

Analyze a HuggingFace repository.

Path Parameters

Parameter Type Description
repo string Repository ID (owner/name)

Query Parameters

Parameter Type Description
dataset boolean Force dataset type
revision string Branch/tag to analyze

Response 200 OK (Determined type)

{
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "is_dataset": false,
  "type": "gguf",
  "type_description": "GGUF Model",
  "file_count": 12,
  "total_size": 4500000000,
  "total_size_human": "4.2 GiB",
  "branch": "main",
  "refs": [
    {"name": "main", "type": "branch", "commit": "abc123..."}
  ],
  "files": [
    {
      "name": "config.json",
      "path": "config.json",
      "size": 1024,
      "lfs": false
    },
    {
      "name": "mistral-7b.Q4_K_M.gguf",
      "path": "mistral-7b.Q4_K_M.gguf",
      "size": 4368438272,
      "lfs": true,
      "sha256": "abc123..."
    }
  ],
  "gguf": {
    "model_name": "Mistral-7B-Instruct-v0.2",
    "quantizations": [
      {
        "name": "Q4_K_M",
        "file": { /* FileInfo */ },
        "quality": 4,
        "quality_stars": "★★★★☆",
        "estimated_ram": 4905066496,
        "estimated_ram_human": "4.6 GiB",
        "description": "Good balance of quality and size"
      }
    ]
  },
  "analyzed_at": "2024-01-15T10:30:00Z"
}

Response 200 OK (Needs selection)

When a repo exists as both model and dataset:

{
  "needsSelection": true,
  "repo": "owner/name",
  "message": "This repository exists as both a model and a dataset. Please select which one you want to analyze.",
  "options": ["model", "dataset"]
}

Examples

# Analyze model
curl http://localhost:8080/api/analyze/TheBloke/Mistral-7B-Instruct-v0.2-GGUF

# Force dataset
curl "http://localhost:8080/api/analyze/facebook/flores?dataset=true"

# Specific revision
curl "http://localhost:8080/api/analyze/owner/repo?revision=v1.0"

Detected Model Types

Type Description Key Fields
gguf GGUF quantized model gguf.quantizations
transformers Transformers model transformers.architecture
diffusers Diffusers pipeline diffusers.pipeline_type
lora LoRA adapter lora.base_model
gptq GPTQ quantized gptq.bits
awq AWQ quantized awq.bits
onnx ONNX model onnx.models
audio Audio model audio.task
vision Vision model vision.task
multimodal Multimodal model multimodal.modalities
dataset Dataset dataset.formats

Cache

GET /api/cache

List cached repositories.

Query Parameters

Parameter Type Description
type string Filter: model, dataset

Response 200 OK

{
  "repos": [
    {
      "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
      "type": "model",
      "path": "/home/user/.cache/huggingface/hub/models--TheBloke--Mistral-7B-GGUF"
    },
    {
      "repo": "facebook/flores",
      "type": "dataset",
      "path": "/home/user/.cache/huggingface/hub/datasets--facebook--flores"
    }
  ],
  "count": 2,
  "cacheDir": "/home/user/.cache/huggingface/hub"
}

Examples

# List all
curl http://localhost:8080/api/cache

# Models only
curl "http://localhost:8080/api/cache?type=model"

GET /api/cache/{repo}

Get cached repository details.

Path Parameters

Parameter Type Description
repo string Repository ID (owner/name)

Response 200 OK

{
  "repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
  "type": "model",
  "path": "/home/user/.cache/huggingface/hub/models--TheBloke--Mistral-7B-GGUF",
  "snapshots": ["main", "v1.0"]
}

Response 404 Not Found

{
  "error": "Repository not found in cache"
}

Example

curl http://localhost:8080/api/cache/TheBloke/Mistral-7B-Instruct-v0.2-GGUF

WebSocket API

Real-time updates via WebSocket connection.

Connection

ws://localhost:8080/api/ws

Connection Parameters

Parameter Value
Read buffer 1024 bytes
Write buffer 1024 bytes
Max message 512 KB
Ping interval 30 seconds
Read timeout 60 seconds
Write timeout 10 seconds

Message Format

All messages are JSON:

{
  "type": "message_type",
  "data": { /* payload */ }
}

Message Types

init

Sent immediately upon connection.

{
  "type": "init",
  "data": {
    "jobs": [ /* all current jobs */ ],
    "version": "3.0.0"
  }
}

job_update

Broadcast when job status changes.

{
  "type": "job_update",
  "data": {
    "id": "a1b2c3d4e5f6",
    "repo": "owner/model",
    "status": "running",
    "progress": {
      "totalFiles": 5,
      "completedFiles": 2,
      "totalBytes": 5000000000,
      "downloadedBytes": 2000000000,
      "bytesPerSecond": 50000000
    },
    "files": [
      {
        "path": "model.bin",
        "totalBytes": 5000000000,
        "downloaded": 2000000000,
        "status": "downloading"
      }
    ]
  }
}

event

General event notifications.

{
  "type": "event",
  "data": {
    "event": "download_complete",
    "jobId": "a1b2c3d4e5f6"
  }
}

JavaScript Example

const ws = new WebSocket('ws://localhost:8080/api/ws');

ws.onopen = () => {
  console.log('Connected');
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);

  switch (msg.type) {
    case 'init':
      console.log('Jobs:', msg.data.jobs);
      break;
    case 'job_update':
      const job = msg.data;
      const percent = (job.progress.downloadedBytes / job.progress.totalBytes * 100).toFixed(1);
      console.log(`${job.repo}: ${percent}%`);
      break;
  }
};

ws.onerror = (err) => console.error('WebSocket error:', err);
ws.onclose = () => console.log('Disconnected');

Python Example

import asyncio
import websockets
import json

async def monitor():
    uri = "ws://localhost:8080/api/ws"
    async with websockets.connect(uri) as ws:
        async for message in ws:
            msg = json.loads(message)
            if msg["type"] == "job_update":
                job = msg["data"]
                progress = job["progress"]
                if progress["totalBytes"] > 0:
                    pct = progress["downloadedBytes"] / progress["totalBytes"] * 100
                    print(f"{job['repo']}: {pct:.1f}%")

asyncio.run(monitor())

Error Handling

Error Response Format

{
  "error": "Error message",
  "details": "Additional details (optional)"
}

HTTP Status Codes

Code Meaning Usage
200 OK Successful GET, existing job
202 Accepted New job created
204 No Content CORS preflight
400 Bad Request Invalid input
401 Unauthorized Auth required/failed
404 Not Found Resource not found
500 Server Error Internal error

Common Errors

Invalid Repository

{
  "error": "Invalid repo format",
  "details": "Expected owner/name"
}

Job Not Found

{
  "error": "Job not found"
}

Analysis Failed

{
  "error": "Analysis failed",
  "details": "404: repository not found"
}

Examples

Complete Download Workflow

# 1. Analyze repository
curl http://localhost:8080/api/analyze/TheBloke/Mistral-7B-Instruct-v0.2-GGUF

# 2. Start download with specific quantization
curl -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{"repo": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "filters": ["q4_k_m"]}'

# 3. Monitor progress
curl http://localhost:8080/api/jobs

# 4. Get specific job
curl http://localhost:8080/api/jobs/a1b2c3d4e5f6

Pause/Resume Workflow

# Start download
JOB_ID=$(curl -s -X POST http://localhost:8080/api/download \
  -H "Content-Type: application/json" \
  -d '{"repo": "owner/large-model"}' | jq -r '.id')

# Pause
curl -X POST "http://localhost:8080/api/jobs/$JOB_ID/pause"

# Resume later
curl -X POST "http://localhost:8080/api/jobs/$JOB_ID/resume"

Real-time Monitoring Script

#!/bin/bash
# monitor.sh - Monitor download progress

JOB_ID=$1

while true; do
  STATUS=$(curl -s "http://localhost:8080/api/jobs/$JOB_ID")
  JOB_STATUS=$(echo "$STATUS" | jq -r '.status')

  if [ "$JOB_STATUS" = "completed" ] || [ "$JOB_STATUS" = "failed" ]; then
    echo "Job $JOB_STATUS"
    break
  fi

  DOWNLOADED=$(echo "$STATUS" | jq '.progress.downloadedBytes')
  TOTAL=$(echo "$STATUS" | jq '.progress.totalBytes')

  if [ "$TOTAL" -gt 0 ]; then
    PCT=$(echo "scale=1; $DOWNLOADED * 100 / $TOTAL" | bc)
    echo "Progress: $PCT%"
  fi

  sleep 2
done

Integration with jq

# Get all running jobs
curl -s http://localhost:8080/api/jobs | jq '.jobs[] | select(.status == "running")'

# Get total download speed
curl -s http://localhost:8080/api/jobs | jq '[.jobs[].progress.bytesPerSecond] | add'

# List repos being downloaded
curl -s http://localhost:8080/api/jobs | jq -r '.jobs[].repo'

CORS

Headers

Header Value
Access-Control-Allow-Origin Configured origins or *
Access-Control-Allow-Methods GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers Content-Type, Authorization
Access-Control-Max-Age 86400

Configuration

CORS origins are configured via server settings (code modification required for custom origins).

Rate Limiting

No rate limiting is implemented. For production deployments, consider using a reverse proxy (nginx, Caddy) with rate limiting.

Security Considerations

  1. Token Protection: HF token is masked in API responses
  2. Directory Lock: Output directories cannot be changed via API
  3. Basic Auth: Optional authentication for all endpoints
  4. CORS: Configurable origin restrictions
  5. Input Validation: Repository format validation
  6. File Size Limits: WebSocket messages limited to 512 KB

See Also