Skip to content

Latest commit

 

History

History
434 lines (347 loc) · 20.8 KB

ROADMAP.md

File metadata and controls

434 lines (347 loc) · 20.8 KB

Workflow Engine Roadmap

Vision

A production-grade, AI-powered workflow orchestration engine with a visual builder UI, dynamic component hot-reload, comprehensive observability, and real-world application examples including a multi-service chat platform.

Phase 1: Foundation (Complete)

Commits: bcf15ee..8e23274

Core Engine

  • Workflow engine with BuildFromConfig, TriggerWorkflow lifecycle
  • HTTP/Messaging/State Machine/Event workflow handlers
  • HTTP/Schedule/Event trigger system
  • Module factory pattern for extensible module types

Modular Framework Migration

  • Switch from GoCodeAlone/modular to CrisisTextLine/modular fork (v1.11.11)
  • Integrate all modular modules: httpserver, httpclient, chimux, scheduler, eventbus, eventlogger, cache, database, auth, jsonschema, reverseproxy

Dynamic Component System (Yaegi)

  • Interpreter pool with sandboxed execution
  • Component registry with lifecycle management
  • File watcher for hot-reload
  • Source validation (stdlib-only imports)
  • Loader with directory/file/string sources

AI-Powered Generation

  • WorkflowGenerator interface with LLM + Copilot SDK backends
  • Anthropic Claude direct API client with tool use
  • GitHub Copilot SDK integration with session management
  • Deploy service bridging AI generation to dynamic components
  • Prompt engineering: system prompt, component prompts, dynamic format

ReactFlow UI

  • Drag-and-drop node palette with categorized modules
  • Property panel for node configuration
  • YAML import/export with round-trip fidelity
  • Undo/redo with history management
  • Validation (local + server)
  • Zustand state management

Test Infrastructure

  • Unit tests: module 73%, ai/llm 85%, dynamic 74%
  • Playwright E2E: app-load, node-operations, connections, import-export, toolbar
  • Vitest component tests: 100 tests across 6 files

Phase 2: Expanded Capabilities (Complete)

Observability Foundation (WS1)

  • MetricsCollector - Prometheus metrics with 6 pre-registered vectors
  • HealthChecker - /health, /ready, /live endpoints
  • RequestIDMiddleware - X-Request-ID propagation with UUID generation

Database Module (WS2)

  • WorkflowDatabase wrapping database/sql with Query/Execute/Insert/Update/Delete
  • DatabaseIntegrationConnector adapter for integration workflows
  • SQL builder helpers (BuildInsertSQL, BuildUpdateSQL, BuildDeleteSQL)

Data Transformation & Webhooks (WS3)

  • DataTransformer with named pipelines (extract, map, filter, convert)
  • WebhookSender with exponential backoff retry and dead letter queue

AI Validation Loop (WS4)

  • Validator with compile-test-retry cycle
  • ValidateAndFix integrating AI regeneration on failure
  • ContextEnrichedPrompt for module/service-aware generation

Dynamic-to-Modular Bridge (WS5)

  • ModuleAdapter wrapping DynamicComponent as modular.Module
  • Configurable provides/requires for dependency injection
  • Engine integration via "dynamic.component" module type

UI Updates

  • 2 new categories: Database, Observability
  • 6 new MODULE_TYPES (29 total)
  • Updated tests for new types/categories

Phase 3: Quality, Testing & Stability (Complete)

Copilot SDK Testing

  • Mock-based unit tests for all Copilot client methods
  • Tool handler invocation tests with realistic payloads
  • Session lifecycle tests (create, send, destroy)
  • Error path coverage (CLI not found, session failure, empty response, malformed JSON)
  • Integration verification with mock Copilot server

E2E Test Expansion

  • Update moduleTypeMap in all e2e specs with 6 new module types
  • Update category count assertions (8 -> 10)
  • New category visibility tests (Database, Observability)
  • Drag-and-drop tests for new module types
  • Property panel tests for new module config fields
  • Complex workflow builder: multi-category, 5+ node workflows
  • Screenshot-driven visual regression for all categories
  • Keyboard shortcuts and accessibility testing

Documentation

  • ROADMAP.md

Phase 4: EventBus Integration (Complete)

PR #18 merged

EventBus Bridge

  • EventBusBridge adapter (MessageBroker -> EventBus)
  • WorkflowEventEmitter with lifecycle events (workflow.started, workflow.completed, workflow.failed, step.started, step.completed, step.failed)

EventBus Trigger

  • EventBusTrigger for native EventBus subscriptions
  • Configure with topics, event filtering, async mode
  • Start/Stop with subscription lifecycle management

Engine Integration

  • Engine integration with workflow/step event emission
  • canHandleTrigger support for "eventbus" trigger type
  • TriggerWorkflow emits start/complete/fail events

UI Updates

  • messaging.broker.eventbus module type in NodePalette (30 total)

Phase 5: AI Server Bootstrap, Test Coverage & E2E Testing (Complete)

PR #19 merged

AI Server Bootstrap (WS1)

  • cmd/server/main.go with HTTP mux and AI handler registration
  • CLI flags for config, address, AI provider configuration
  • Graceful shutdown with signal handling
  • initAIService with conditional Anthropic/Copilot provider registration
  • cmd/server/main_test.go with route verification tests

Go Test Coverage (WS2)

  • Root package (engine_test.go): 68.6% -> 80%+
  • Module package: 77.1% -> 80%+
  • Dynamic package: 75.4% -> 80%+
  • AI packages: maintain 85%+

Playwright E2E Tests (WS3)

  • Shared helpers (helpers.ts) with complete module type map
  • deep-module-coverage.spec.ts: All 30 module types verified
  • deep-complex-workflows.spec.ts: Multi-node workflow tests
  • deep-property-editing.spec.ts: All field types tested
  • deep-keyboard-shortcuts.spec.ts: Shortcut verification
  • deep-ai-panel.spec.ts: AI Copilot panel tests
  • deep-component-browser.spec.ts: Component Browser tests
  • deep-import-export.spec.ts: Complex round-trip tests
  • deep-edge-cases.spec.ts: Edge case coverage
  • deep-accessibility.spec.ts: A11y testing
  • deep-toast-notifications.spec.ts: Toast behavior tests
  • deep-visual-regression.spec.ts: Visual regression baselines

Phase 6: Integration, Real-World Applications & Production Infrastructure (Complete)

Module Expansion (48 Total Module Types)

  • HTTP modules (10): http.server, http.router, http.handler, http.middleware.auth/cors/logging/ratelimit/requestid, http.proxy, http.simple_proxy
  • Messaging modules (6): messaging.broker, messaging.broker.eventbus, messaging.handler, messaging.nats, messaging.kafka, notification.slack
  • State machine modules (4): statemachine.engine, state.tracker, state.connector, processing.step
  • Modular framework modules (10): httpserver, httpclient, chimux, scheduler, auth, eventbus, cache, database, eventlogger, jsonschema
  • Storage/persistence modules (4): database.workflow, persistence.store, storage.s3, static.fileserver
  • Observability modules (3): metrics.collector, health.checker, observability.otel
  • Data/integration modules (4): data.transformer, api.handler, webhook.sender, dynamic.component
  • Auth modules (2): auth.jwt, auth.modular
  • Reverse proxy modules (2): reverseproxy, http.proxy
  • Trigger types (5): http, schedule, event, eventbus, mock

Order Processing Pipeline Example

  • 10+ module YAML config with HTTP servers, routers, handlers, data transformers, state machines, brokers, and observability
  • End-to-end pipeline demonstrating module composition

Chat Platform Example (73 files)

  • Multi-service Docker Compose architecture (gateway, API, conversation, Kafka, Prometheus, Grafana)
  • 18 dynamic components: ai_summarizer, pii_encryptor, survey_engine, risk_tagger, conversation_router, message_processor, keyword_matcher, escalation_handler, followup_scheduler, data_retention, notification_sender, webchat_handler, twilio_provider, aws_provider, partner_provider
  • Full SPA with role-based views: admin, responder, supervisor dashboards
  • Queue health monitoring with per-program metrics
  • Conversation state machine (queued -> assigned -> active -> wrap_up -> closed)
  • Real-time risk assessment with keyword pattern matching (5 categories: self-harm, suicidal-ideation, crisis-immediate, substance-abuse, domestic-violence)
  • PII masking in UI (phone numbers, identifiers)
  • Webchat widget for web-based texters
  • Seed data system (users, affiliates, programs, keywords, surveys)
  • Architecture docs, user guide, and screenshots

JWT Authentication

  • JWTAuthModule with user registration, login, token generation/validation
  • Seed file loading with bcrypt password hashing
  • Role-based metadata in JWT claims and login response
  • Auth middleware integration for endpoint protection

PII Encryption at Rest

  • AES-256-GCM FieldEncryptor with SHA-256 key derivation
  • Configurable field-level encryption (encrypt specific fields, leave others plaintext)
  • Integration with PersistenceStore (encrypt on save, decrypt on load)
  • Integration with KafkaBroker (encrypt/decrypt Kafka message payloads)
  • 15 sub-tests covering encrypt/decrypt, round-trip, key rotation, edge cases

REST API Handler Enhancements

  • View handler pattern: sourceResourceName + stateFilter for cross-resource views
  • Queue health endpoint with per-program aggregation
  • Sub-action support for nested resources (e.g., /conversations/{id}/messages)
  • Inline risk assessment on message append and conversation creation

Docker Multi-Stage Builds

  • Dockerfile for chat-platform (builder/runtime stages, Alpine Linux, CGO_ENABLED=0)
  • Dockerfile for ecommerce-app
  • Docker Compose with health checks, volume mounts, and service dependencies

Observability Stack

  • Prometheus metrics collection with custom dashboard
  • Grafana dashboards for chat platform monitoring
  • Health check endpoints integrated with Docker orchestration

Phase 7: Quality, Documentation & Production Readiness (Complete)

Dynamic Field Mapping

  • FieldMapping type with fallback chains and primary/resolve/set operations
  • Schema-agnostic field mapping for REST API handler modules (42+ references refactored)
  • Runtime field resolution from workflow context via FieldMapping.Resolve()
  • Configurable field aliases in YAML (fieldMapping, transitionMap, summaryFields)
  • Engine integration: fieldMapping/transitionMap/summaryFields wired from YAML config
  • 18 unit tests for FieldMapping type
  • Component-level field contracts: FieldContract type, ContractRegistry, pre-execution validation
  • 4 chat platform components updated with contracts (keyword_matcher, conversation_router, escalation_handler, ai_summarizer)

YAML Config Validation

  • JSON Schema generation from WorkflowConfig structs (schema/ package)
  • Validation at load time with descriptive error messages (integrated into BuildFromConfig)
  • Schema export endpoint (GET /api/schema)
  • 41 tests covering schema generation, validation rules, and HTTP endpoint

Handler Test Coverage

  • IntegrationWorkflowHandler: database connector path, nil/stale connector
  • ExecuteIntegrationWorkflow: retry logic, variable substitution, error handlers
  • ExecuteWorkflow: multi-step dispatch
  • Service helper edge cases (FixMessagingHandlerServices, PatchAppServiceCalls)
  • 29 new tests, integration.go/service_helper.go/app_helper.go at 100%

Performance & Scalability

  • Interpreter pool benchmarks (creation ~2.4ms, execute ~1.5us, pool contention negligible)
  • Concurrent workflow stress tests (100+ concurrent, ~28K workflows/sec, zero goroutine leaks)
  • UI rendering performance with 50+ nodes (Playwright E2E)

Deployment

  • Helm chart for Kubernetes (monolith/distributed modes, HPA, ServiceMonitor)
  • Configuration via environment variables (WORKFLOW_CONFIG, WORKFLOW_ADDR, etc.)
  • CI/CD pipelines (test matrix, Docker build, Helm lint, release workflow)
  • Multi-stage Dockerfile

Security Hardening

  • Input validation middleware (size limits, content-type, JSON well-formedness)
  • Dynamic component resource limits (execution timeout, output size)
  • Rate limiting with IP/token/combined strategies and stale cleanup
  • Audit logging with structured JSON (auth, admin, escalation, data access events)

Documentation

  • README.md rewrite with 48 module types, chat platform, full feature set
  • CHANGELOG.md with Phase 2-6 entries
  • API documentation for REST endpoints (docs/API.md)

Phase 8: Advanced Features & Ecosystem (Complete)

Multi-Chat & Collaboration UI

  • Responder-to-Responder direct messaging (DM threads with real-time polling)
  • Supervisor-to-Responder real-time chat (shared DM system)
  • Shared resource panel (15 canned responses, search/filter, copy/insert)
  • Conversation transfer with live chat handoff (visual indicators, history)

Plugin Ecosystem

  • Plugin registry with manifest validation and HTTP CRUD API
  • Component versioning with semver constraints and compatibility checking
  • Plugin SDK with template generator and documentation generator
  • Community submission validator with 8 checks and review checklist

Advanced AI Integration

  • Response suggestion engine with LLM + template fallback and caching
  • Conversation classifier with 4 categories, priority scoring, 14 rules
  • Sentiment analysis with lexicon fallback, trend detection, sharp-drop alerts
  • Supervisor alert engine with hybrid rule-based + AI detection, 4 alert types

Multi-Tenancy & Scale

  • Worker pool with auto-scaling and consistent hash partitioning
  • Per-tenant quota enforcement with token bucket rate limiting
  • Multi-region routing with data residency compliance
  • Generic connection pool and LRU cache with TTL

Observability & Operations

  • OpenTelemetry distributed tracing (HTTP spans, workflow spans, context propagation)
  • 3 Grafana dashboards (workflow overview, chat platform, dynamic components)
  • 7 Prometheus alerting rules with runbooks
  • SLA monitoring with uptime/latency/error budget tracking

Phase 9: Production Hardening & Developer Experience (Complete)

Testing & Reliability

  • Integration test suite (12 E2E tests: config loading, lifecycle, workflow execution, module wiring)
  • Chaos testing (6 tests: random component failures, concurrent chaos, recovery validation)
  • Load testing (6 tests: ~27K workflows/sec, concurrent dispatch, cache/pool performance)
  • Regression testing (10 tests: validates all 34 example YAML configs load correctly)

Developer Experience

  • CLI tool wfctl with validate, inspect, run, plugin, schema subcommands
  • Interactive workflow debugger with breakpoints, step/continue, variable inspection, HTTP API
  • Plugin development hot-reload watcher (file system monitoring, auto-reload on change)

Platform Features

  • Webhook retry with exponential backoff, max retries, dead letter store with HTTP endpoints
  • Cron scheduler with job CRUD, expression parsing, concurrent execution control
  • Workflow versioning with version store, rollback, diff comparison, HTTP API
  • Environment promotion pipeline (dev -> staging -> prod) with approval gates

Security & Compliance

  • OAuth2/OIDC provider with token validation, auth code flow, JWKS endpoint
  • RBAC middleware with permission model (action + resource), role definitions, enforcement
  • Compliance reporting for SOC2 and HIPAA (23 controls, evidence collection, PDF-ready output)
  • Secret management with provider interface (env, file, Vault stub), secret:// URI resolver

Documentation & Community

  • OpenAPI 3.0 specification (2245 lines, 65+ endpoints, full schema definitions)
  • Tutorial series: getting started, building plugins, scaling workflows, chat platform walkthrough
  • 5 Architecture Decision Records (YAML config, modular framework, Yaegi, hybrid AI, field contracts)
  • CONTRIBUTING.md with development setup, PR process, coding standards
  • CODE_OF_CONDUCT.md (Contributor Covenant v2.1)

Phase 10: Multi-Affiliate Routing & End-to-End QA (Complete)

Multi-Affiliate Conversation Routing

  • JWT claims enrichment with affiliateId/programIds from user metadata
  • handleGetAll() filtering by affiliate/program query params with JWT defaults
  • Conversation router with real routing logic (keyword → program → affiliate mapping)
  • Queue health scoping by affiliate (non-admin users see only their affiliate's data)

Conversation Data Fixes

  • Initialize messages array on conversation creation (fix empty chat views)
  • Ensure messages sub-action handler creates slice before appending

SPA Multi-Tenant Updates

  • Responder view: pass affiliate/program params in API calls
  • Supervisor view: filter users and conversations by affiliate
  • Queue view: scope queue health by affiliate
  • Chat view: filter transfer responder list by affiliate
  • Display program/affiliate context badges in conversation cards

Seed Data Enrichment

  • Add EU-West users (responder + supervisor for aff-003/prog-004)
  • Add PARTNER keyword for prog-004

End-to-End QA Testing

  • Playwright multi-agent QA: texters send keywords, verify routing to correct programs
  • Cross-affiliate isolation: responders see only their affiliate's conversations
  • Supervisor scoping: supervisors see only their affiliate's responders
  • Multi-message flow verification with screenshots
  • Queue health per-affiliate validation

Phase 11: End-to-End QA & Polish (Complete)

State Machine Bug Fixes

  • Fix instance ID mismatch (double "conv-" prefix between webhooks-api and conversations-api)
  • Fix field name normalization (Twilio "Body" to lowercase "body" for contract validation)
  • Fix auto-transition state sync (assigned to active not reflected in resource data)
  • Fix bridgeToConversation initial state (was hardcoding "queued", now uses "new")

Playwright E2E Tests for Chat Platform

  • Login flow: verify all 8 seed users can log in with correct roles
  • Conversation routing: send messages with HELLO/TEEN/WELLNESS/PARTNER keywords, verify routing
  • Cross-affiliate isolation: login as aff-001 responder, verify only aff-001 conversations visible
  • Supervisor view: verify supervisors see only their affiliate's responders and conversations
  • Message flow: send messages both directions, verify real-time updates
  • Multi-chat: open multiple conversations simultaneously, verify messages route correctly
  • Queue health: verify per-affiliate program stats display correctly
  • Transfer flow: transfer conversation between responders, verify handoff
  • Escalation flow: test medical/police escalation state transitions
  • Screenshot documentation: 21 QA screenshots captured

UI Improvements

  • Accept Conversation banner for queued conversations in chat view
  • Tags sidebar auto-refresh after applying tags

Platform Polish

  • Error handling improvements for edge cases discovered during QA
  • Performance profiling of conversation routing under load (~20ms creation, ~6ms queries)
  • Documentation updates based on QA findings

Module Type Summary

Category Count Types
HTTP 10 http.server, http.router, http.handler, http.middleware.{auth,cors,logging,ratelimit,requestid}, http.proxy, http.simple_proxy
Messaging 6 messaging.broker, messaging.broker.eventbus, messaging.handler, messaging.nats, messaging.kafka, notification.slack
State Machine 4 statemachine.engine, state.tracker, state.connector, processing.step
Modular Framework 10 httpserver, httpclient, chimux, scheduler, auth, eventbus, cache, database, eventlogger, jsonschema
Storage/Persistence 4 database.workflow, persistence.store, storage.s3, static.fileserver
Observability 3 metrics.collector, health.checker, observability.otel
Data/Integration 4 data.transformer, api.handler, webhook.sender, dynamic.component
Auth 2 auth.jwt, auth.modular
Reverse Proxy 2 reverseproxy, http.proxy
Total 48

Trigger Types

Type Description
http HTTP request triggers
schedule Cron-based scheduled triggers
event Generic event triggers
eventbus Native EventBus subscription triggers
mock Test/mock triggers

Example Configs

The example/ directory contains 37+ YAML configurations demonstrating different workflow patterns, plus two full application examples:

Example Description
order-processing-pipeline.yaml E-commerce pipeline with 10+ modules across 5 categories
chat-platform/ Production-grade mental health chat platform with 73 files, multi-service Docker Compose, 18 dynamic components, full SPA