Make Relayfile completely provider-agnostic

[khaliqgant]

Summary

Transform Relayfile from a Notion-specific webhook processor into a pure generic webhook-to-VFS engine. All provider-specific logic (including Nango integration) lives outside the Relayfile repo.

Changes

Phase 1: Generic API Exposure (Non-Breaking)

• ✅ Add POST /v1/webhooks/ingest - Accept generic webhook envelopes from any provider
• ✅ Add GET /v1/writeback/pending - List pending writeback items for external consumption
• ✅ Add POST /v1/writeback/{id}/ack - Acknowledge writeback with status updates
• ✅ Updated OpenAPI spec with endpoint schemas and provider-agnostic examples
• ✅ Comprehensive test coverage for all new endpoints

Phase 2: Optional Adapters (Low Risk)

• ✅ Implement ParseGenericEnvelope() for provider-agnostic webhook parsing
• ✅ Fallback to generic parser when no provider-specific adapter exists
• ✅ Remove hardcoded "notion" from buildAdaptersFromEnv()
• ✅ Support any provider through generic envelope format

Phase 3: Complete Notion Removal

• ✅ Delete all Notion-specific files:
  • internal/relayfile/notion_http_client.go
  • internal/relayfile/notion_http_client_test.go
  • internal/relayfile/notion_writeback_test.go
  • scripts/import-notion-pages.sh
• ✅ Remove NotionAdapter and related types from adapters.go
• ✅ Remove buildNotionTokenProvider functions
• ✅ Remove all Notion-specific tests

Critical Fixes

• ✅ Fix store.go syntax corruption (inferProviderFromPath, fallbackProviderPath)
• ✅ Fix /webhooks/ingest payload to preserve event_type/path in envelope
• ✅ Make delivery_id optional with auto-generation
• ✅ Implement AcknowledgeWriteback with proper operation lookup
• ✅ Add snapshot support to inMemoryWritebackQueue
• ✅ Align OpenAPI schema with implementation (opStatus removed, data optional)

Documentation Updates

• ✅ Update README with generic provider examples
• ✅ Update compose.env.example (remove all NOTION_* variables)
• ✅ Update architecture-ascii.md to show external provider integration
• ✅ Update production-validation.md with generic webhook examples
• ✅ Update relayfile-v1-spec.md with generic provider references
• ✅ Update sdk/relayfile-sdk/README.md with generic examples
• ✅ Update send-internal-envelope.sh to be provider-agnostic
• ✅ Update dashboard.go defaults from /notion to /
• ✅ Update openapi examples from notion to salesforce/slack

Architecture

Before (Notion-Specific):

Notion API ←→ RelayFile (calls Notion APIs) ←→ Postgres

After (Provider-Agnostic):

External Providers
        ↓
    Nango / relay-cloud (external service)
        ↓
POST /v1/webhooks/ingest (generic format)
        ↓
    RelayFile VFS
        ↓
GET /v1/writeback/pending + POST /v1/writeback/{id}/ack
        ↓
    External Service (polls queue, calls providers)

Testing

• ✅ Phase 1: New endpoint tests (generic webhook ingest, writeback pending, ack)
• ✅ Phase 2: Generic envelope parsing tests (multiple providers)
• ✅ Phase 3: Removed Notion-specific test files
• ✅ Code review: Fixed critical compile errors and schema mismatches
• ✅ Final scan: Only test fixtures retain Notion references (appropriate for testing)

Notes

• No breaking changes to existing filesystem APIs
• All provider integration now handled externally via generic webhook API
• Writeback queue is externally consumed and acknowledged
• Fully provider-agnostic; tested with examples for multiple providers

🚀 Generated with Claude Code

---

[devin-ai-integration]

Devin Review found 3 potential issues.

View 6 additional findings in Devin Review.

[devin-ai-integration]

🔴 Merging data fields can overwrite validated event_type and path in envelope payload

In handleGenericWebhookIngest, the code first validates and sets event_type and path from the top-level request fields into envelopePayload, then blindly merges all keys from the data sub-object into the same map. If a caller includes event_type or path keys inside their data object, those will silently overwrite the validated values.

Root Cause and Impact

At internal/httpapi/server.go:1756-1764, the envelope payload is constructed:

envelopePayload["event_type"] = eventType  // validated
envelopePayload["path"] = path              // validated
// ...
for k, v := range d {                       // from data
    envelopePayload[k] = v                   // overwrites!
}

Downstream, ParseGenericEnvelope at internal/relayfile/adapters.go:41-51 reads event_type and path from req.Payload, so the overwritten values are used for processing. A request like {"provider":"x", "event_type":"file.updated", "path":"/safe/path", "data":{"event_type":"file.deleted", "path":"/other/path"}} would bypass the HTTP-layer validation and cause a delete at /other/path instead of an update at /safe/path.

Impact: Bypasses input validation; can cause incorrect file operations (e.g., deleting a file when an update was intended, or operating on the wrong path).

Suggested change

	if d, ok := payload["data"].(map[string]any); ok {
	for k, v := range d {
	envelopePayload[k] = v
	}
	if d, ok := payload["data"].(map[string]any); ok {
	for k, v := range d {
	if k == "event_type" || k == "path" {
	continue
	}
	envelopePayload[k] = v
	}
	}

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔴 Webhook ingest POST endpoint requires only sync:read scope instead of a write scope

The POST /v1/workspaces/{workspaceId}/webhooks/ingest endpoint requires only sync:read scope, but it performs a write operation (ingesting webhook data into the queue). This is inconsistent with the existing pattern and a security concern.

Detailed Explanation

At internal/httpapi/server.go:178-180:

case len(parts) == 5 && parts[3] == "webhooks" && parts[4] == "ingest" && r.Method == http.MethodPost:
    requiredScope = "sync:read"
    route = "generic_webhook_ingest"

All other POST endpoints in the sync domain use sync:trigger as the required scope. For example:

• sync_dead_letter_ack (POST) → sync:trigger (internal/httpapi/server.go:161)
• sync_dead_letter_replay (POST) → sync:trigger (internal/httpapi/server.go:164)
• sync_refresh (POST) → sync:trigger (internal/httpapi/server.go:167)
• writeback_ack (POST) → sync:trigger (internal/httpapi/server.go:185)

Using sync:read means any client with a read-only token can inject arbitrary webhook data, create files, and queue processing jobs.

Impact: Clients with read-only tokens can perform write operations through the webhook ingest endpoint, violating the principle of least privilege.

Suggested change

	requiredScope = "sync:read"
	requiredScope = "sync:trigger"

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🔴 Race condition in writeback queue causes phantom pending items in snapshot

In inMemoryWritebackQueue, items are added to the items map after being sent to the channel. A concurrent Dequeue can process and delete the item from items before it's inserted, leaving a stale entry that never gets cleaned up.

Root Cause and Impact

In TryEnqueue at internal/relayfile/store.go:541-544:

case q.ch <- task:          // 1. task sent to channel
    q.mu.Lock()
    q.items[task.OpID] = task  // 3. added to items map (too late)
    q.mu.Unlock()

Meanwhile in Dequeue at internal/relayfile/store.go:571-574:

case task := <-q.ch:         // 2. task received from channel
    q.mu.Lock()
    delete(q.items, task.OpID) // delete is a no-op since insert hasn't happened
    q.mu.Unlock()

If the worker dequeues between steps 1 and 3, the delete in Dequeue is a no-op (item not yet in map), and then step 3 inserts it permanently. The item remains in q.items forever because nothing will delete it again.

SnapshotWritebacks() reads from q.items, so GetPendingWritebacks will return these phantom items to external consumers, who will see ghost entries for already-processed writebacks.

Impact: Memory leak in the items map; external consumers polling /writeback/pending may see and attempt to ACK already-completed writebacks.

Prompt for agents

In internal/relayfile/store.go, the inMemoryWritebackQueue has a race condition where items are added to the items map after being sent to the channel. The fix should ensure the item is added to the items map BEFORE being sent to the channel in both TryEnqueue (lines 536-549) and Enqueue (lines 551-564). Similarly, in Dequeue (lines 566-579), the delete from items should happen AFTER receiving from the channel but that part is already correct. The key change is: in TryEnqueue and Enqueue, acquire q.mu.Lock(), add to q.items, then send to channel. If the channel send fails (in TryEnqueue's default case), remove from q.items before returning false. For Enqueue's ctx.Done case, also remove from items.

---

Was this helpful? React with 👍 or 👎 to provide feedback.