Extract core, separate from UI, refactor data models and pipeline. Almost a full rewrite.#3
Closed
Extract core, separate from UI, refactor data models and pipeline. Almost a full rewrite.#3
Conversation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…n files Move state management from App.tsx (~3500 lines) into two Zustand stores: - conversation-store.ts: conversations, selections, workflow actions - ui-store.ts: dialog state, presets, import state, dimensions Extract workflow pipeline from App.tsx into domain-organized files: - parse.ts, count-tokens.ts, segment.ts: pre-AI workflow steps - component-identification.ts: discover component list per dimension - component-classification.ts: classify each part into a component - color.ts: assign colors to components - summarize.ts, analyze.ts: AI summary and analysis generation - pipeline.ts: event dispatcher with per-event handlers - runner.ts, types.ts, dimensions.ts: shared infrastructure App.tsx shrinks to ~1150 lines of JSX, effects, and handler wiring. Each workflow domain file is a vertical slice (activity + step runner). The pipeline dispatcher replaces a 390-line monolithic function with a 20-line switch and 9 self-contained event handlers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the WorkflowRunner class and Activity<T> type. Replace with: - Notify: plain callback type (id, update) => void - startStep/endStep/updateState/markComplete: free functions - timed(): generic async timing wrapper Reorganize workflow files by domain instead of abstraction layer: - parse.ts, count-tokens.ts, segment.ts, color.ts, summarize.ts, analyze.ts each contain the activity logic + step runner as a vertical slice - component-identification.ts: discover component list (calls identifyComponents) - component-classification.ts: classify parts (calls mapComponentsToIds) - pipeline.ts: event dispatcher + composite sequences Each domain file calls the core lib directly — no intermediate Activity<T> wrapper or class instantiation needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Plan 1 - Componentisation Cleanup: Split componentisation.ts into component-types, component-identification, component-classification, component-coloring. Rename static-componentisation to static-components. Replace getComponentisationConfig with direct getAIConfig. Rename prompt key "component-mapping" to "component-classification". Plan 2 - Simplified Pipeline: Replace WorkflowEvent enum (9 values) with PipelineStep enum (6 ordered values). Replace 8 handler functions + dispatcher with runPipelineFrom(startStep). Summary/analysis become standalone on-demand functions. Plan 3 - First-Class Dimensions: Remove legacy top-level component fields (components, componentMapping, componentTimeline, componentColors, targetDimension) from WorkflowState. dimensions is now the single source of truth. Add accessor helpers (getDimension, getDefaultDimension, getAllComponents). Plan 4 - First-Class Groups: Groups become lightweight metadata (Group type with name + fileIds). Stored in separate groups store slice, not as concatenated WorkflowStates. Remove isGrouped, sourceConversations, messageSourceMap from WorkflowState. Plan 5 - Store Decomposition: Extract buildBaseContext to workflow/context.ts, file drop parsing to lib/file-import.ts, exportPromptsAsPreset to lib/export-builder.ts, orchestration functions to workflow/orchestrate.ts. Store shrinks from 788 to 332 lines. Additional: Rename SourceInfo→OriginInfo, messageSourceMap→messageOriginMap (provenance). Rename sourceConversations→memberFiles, sourceWorkflowStates→memberWorkflowStates. Define Source discriminated union type (FileSource | GroupSource) in source-types.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allows turning off or adjusting reasoning/thinking for OpenAI reasoning models (gpt-5 series, o-series) via VITE_AI_THINKING=none|low|medium|high. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Store methods no longer accept setSelectedId or selectedId parameters. They just mutate data and return intent (e.g., groupConversations returns the group ID). App.tsx handles all navigation reactively: - useEffect auto-selects when selectedId is null or invalid - Wrapper functions in App.tsx call setSelectedId after store returns - No more stale closure captures from passing selectedId into async functions This fixes the bug where conversations would disappear after processing (caused by setSelectedId being called too late, after workflows completed, with a stale selectedId value). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract messageOriginMap into separate groupDisplayData memo (remove as-any cast) - Fix WorkflowDetailModal opening for group entries in sidebar - Strengthen color prompt to enforce named colors only (no hex codes) - Apply-prompts-to-all now also copies component list and colors as presets - Enable summary/analysis generation for groups (stored on Group object) - Virtual group state includes aiSummary/analysis from Group Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add VITE_AI_BASE_URL and VITE_AI_API_MODE env vars to allow switching between OpenAI and any OpenAI-compatible provider. Centralize model creation in ai-config.ts via createModel() helper, removing duplicate createOpenAI() calls across all pipeline files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs now show [identifying-components] and [classifying-components] as distinct phases instead of a single [finding-components]. Step timings record each separately. The UI progress indicator still shows one combined "Find components" step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Small cycle button above waffle chart in Components tab. Setting is shared across WaffleChart, StackedBarChartView, and ComponentComparisonView via UI store. Default is 0 (unchanged behavior). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ComparisonLegend is a separate function component that didn't have access to the percentPrecision prop. Now receives it explicitly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Step runners (identify, classify, color) accept optional dimNames[] to process only specific dimensions instead of all - runPipelineFrom passes dimNames through to step runners - Extract runDimensionSteps to eliminate duplication across processNewFile, resumeFromPause, and runPipelineFrom - Move customPrompt, customColoringPrompt, customComponents from WorkflowState top-level into DimensionData (dimensions are now self-contained) - Rewrite applyPromptsToAll to diff per-dimension prompts and only reprocess dimensions that actually changed - App.tsx passes [dimName] when editing prompts/components/colors for a single dimension Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add reprocessTarget() to orchestrate.ts: resolves group-or-file,
fans out to member files for groups, processes single file otherwise
- Replace all if (selectedGroup) { loop } blocks in App.tsx handlers
with calls to handleReprocessTarget
- Coloring prompt reprocessing now works for groups (was missing)
- Remove stale fields from Group type: customPrompt,
customColoringPrompt, customSegmentationPrompt, segmentationThreshold
(component prompts live on member files' dimensions, not the group)
- Remove unused runPipelineFrom import from App.tsx
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Components now read from Zustand stores directly instead of receiving 30+ callback props threaded through App.tsx. This eliminates the controlled+fallback dual-state pattern and makes each component self-contained. New files: - stores/url-store.ts: URL state as a Zustand store (replaces useUrlState hook) - hooks/useWorkflowActions.ts: extracted handler functions from App.tsx - lib/message-filters.ts: shared filter logic (was duplicated 3x) Key changes: - ConversationList: 36 props → 0 (reads stores directly) - ConversationView: removed 24 URL-state props and dual-state pattern - ComponentComparisonView: removed 14 props and 6 local state shadows - App.tsx: 1278 → 632 lines (layout shell + effects + dialogs only) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pipeline (pipeline.ts) — just the step chain: - runDimensionSteps: Segment → Identify → (Classify + Color in parallel) - processNewFile: Parse → CountTokens → Static → runDimensionSteps - resumeFromPause: runDimensionSteps from Segment Orchestration (orchestrate.ts) — who runs what, where results go: - runPipelineFrom, reprocessWithRunner, reprocessTarget (moved from pipeline) - runWorkflows batch processing (moved from pipeline) - generateSummaryOnDemand/generateAnalysisOnDemand/rerunSummary (moved) - New store-level functions: generateAnalysisForTarget, generateSummaryForTarget, rerunSummaryForTarget — handle group-vs-file dispatch with group-aware notify routing UI layer (useWorkflowActions.ts) — thin wrappers only: - All handlers now call store actions, never pipeline functions directly - Removed buildBaseContext, Notify, runPipelineFrom imports - Group fan-out handled by orchestrate, not UI code Also: - Rename runner.ts → notify.ts (it's progress/state helpers, not a runner) - Classify + Color run in parallel after Identify (both only need component list, not each other) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WorkflowDetailModal now reads conversation data from the store and calls action functions directly instead of receiving ~20 callback props. - Props reduced from 28 to 3 (isOpen, onClose, conversationId) - Reads conv data, group info, member files from useConversationStore - Calls openPromptEditor, generateAnalysis, etc. from useWorkflowActions - ConversationList call site simplified from 28 props to 3 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ai-config.ts: LanguageModelV2 → LanguageModel (renamed in ai SDK) - GroupFileOrderEditor: optional chaining on possibly-undefined array access - url-fetch.ts: non-null assertion on array element - workflow/types.ts: widen stepTimings to accept sub-step keys - claude-transcripts-parser: type assertions for Zod union narrowing (ClaudeContent catch-all schema prevents TS discriminated union narrowing) - codex-transcripts-parser: same Zod union narrowing fix - export-import tests: remove null analysis values (should be undefined) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- WorkflowDetailModal: derive dimensions from first member file for groups (was undefined because groups aren't in conversations array) - openPromptEditor/openComponentsEditor/openColoringPromptEditor: for groups, read current prompt from first member file - applyPrompt/applyComponents/applyColoringPrompt: for groups, update all member files' dimensions in the store (was only updating by group ID which matched nothing) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The diff was only comparing prompts and customComponents between source and target dimensions. When both had undefined prompts (default), nothing would run even if the actual component lists were different. Now compares the actual components array too, so "apply to all" correctly reprocesses targets whose components differ from the source. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add "segments-asc" and "segments-desc" sort options that flatten messages into individual parts sorted by token count - Change default coloring prompt to request hex codes with a reference palette, instead of restricting to named color strings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sion data Both runClassifyComponents and runAssignColors run in parallel and were replacing dims[dimName] with a spread of the original object. Whichever finished last would wipe the other's work, causing either missing componentMapping (no waffles) or missing componentColors. Fix: mutate the existing dimData object in place instead of replacing it. Also add store/dimensions to __debug window object for easier debugging. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each dimension-level step (Identify, Classify, Color) now checks its inputs against its existing outputs and skips if they match, rather than relying on the orchestrator to decide what to run. This removes the branching logic from applyPromptsToAll (segChanged/identifyDims/ colorDims) — it now copies all state from source and runs the full pipeline, letting each step skip itself. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The expandedDimension state was keyed by dimension name alone, so expanding "default" on one conversation expanded it everywhere. Now keyed by "conversationId:dimName" so each is independent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move selection state and API key state from conversation-store to ui-store. Extract export actions as standalone functions. Remove 7 orchestration passthrough methods from conversation-store — hooks now call orchestrate functions directly via an external accessor, making conversation-store a pure data/CRUD layer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reorganize the codebase into a clear layered directory structure: - model/ — domain types and schemas (schema, types, dimensions, export-schema, presets) - operations/ — pure transforms (aggregation, token-counting, static-components, etc.) - parsers/ — pluggable input format adapters - stages/ — processing pipeline stages with ai/ infrastructure nested inside - pipeline/ — orchestration (step sequencing, notify, orchestrate) - stores/ — state management + actions (moved useWorkflowActions here) - ui/ — React components, hooks, and UI-specific lib files - lib/ — generic utilities (id-generator) Each algorithm file and its workflow wrapper are merged into one file per stage. The dependency rule is enforced: model → nothing, operations → model, parsers → model, stages → model + operations, pipeline → stages, stores → pipeline, ui → stores. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move browser I/O (downloadExport, exportPromptsAsPreset) from operations/ to ui/lib/export-download.ts - Move workflow-logger and stage-logger from stages/ai/ to pipeline/ (they're cross-cutting infrastructure, not AI-specific) - Replace ProcessingPhase/ProcessingStep with unified Stage and StageGroup types; consistent -ing verb forms throughout - Rename Workflow* types to Pipeline* (PipelineState, PipelineCallbacks, PipelineOptions, etc.) to match directory naming - Rename DimensionData.components to discoveredComponents; add getEffectiveComponents() helper to consolidate override logic - Add I/O concern note to stages/ai/preset-loader.ts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Write docs/architecture.md: directory structure, layers, dependency rules, stage descriptions, pipeline flow, and key concepts - Archive 12 completed/historical plan docs to docs/archive/ - Update CAPABILITIES.md file locations to new structure - Update tech-stack.md: useState → Zustand - Update system-overview.md and dimension-scoped-pipeline-design.md with cross-references to architecture.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each dimension-level stage (identify, classify, color) now operates on a single DimensionData instead of internally looping over all dimensions. The pipeline owns the iteration: - identifyForDimension(conversation, dimData, config, id) - classifyForDimension(conversation, dimData, config, id) - colorForDimension(dimData, config, id, presetColors?) pipeline/pipeline.ts handles: which dimensions to process, parallelism (classify + color run in parallel per dimension), error collection, and timing. Stages are now simpler — pure single-dimension logic with no orchestration concerns. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The dimension loop in pipeline/pipeline.ts silently skipped all work when dims["default"] didn't exist (new files start with empty dimensions). Added ensureDimension() and createEmptyDimension() to model/dimensions.ts as the canonical way to initialize dimensions. Replaced 7 inline empty DimensionData literals across stores/actions, pipeline/orchestrate, and ui/App with the factory function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Stages now return results instead of mutating ctx/dimData. The pipeline is defined as a PIPELINE array of StageDescriptors, and a runner handles sequencing, dimension loops, parallelism, logging, timing, and store updates. Classify+color parallel execution is race-free by construction — results are merged after both complete. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the declarative PIPELINE array + runner with an imperative runPipeline function that reads top-to-bottom like pseudocode. Absorb all orchestration (reprocessTarget, applyPromptsToAll, batch processing, on-demand summary/analysis) into pipeline.ts and delete orchestrate.ts entirely. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each stage checks if its work is already done and skips if so. Callers signal "re-run this stage" by clearing its outputs (e.g., dim.discoveredComponents = [] to force re-identify). Delete PipelineStep enum — no longer needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Delete processNewFile and resumeFromPause — they were identical try/catch wrappers around runPipeline. Error handling is now inside runPipeline itself. - Remove redundant markComplete calls from reprocessTarget and applyPromptsToAll — the pipeline's final updateState already pushes the complete state. - Extract makeGroupAwareCallbacks to deduplicate streaming callback setup for summary/analysis. - Fix duplicate ctx.aiSummary = "" in rerunSummaryForTarget. - Simplify resumePipelinesWithApiKey to call runPipeline directly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Session recorder (src/lib/session-recorder.ts): - recordCall() wraps functions to capture args, return values, errors, timing - Captures Zustand store snapshots (conversations, groups) before/after calls marked with captureStore: true - Safe serialization handles File, Map, Set, circular refs, large strings - Session start/stop via window.__session.start() / .stop() in browser console - On stop, downloads a JSON file with all recorded call entries - Zero overhead when not recording (single boolean check) Instrumented functions (33 total): - Pipeline (9): run, runPipeline, reprocessTarget, runPipelineMutation, applyPromptsToAll, generateAnalysisForTarget, generateSummaryForTarget, rerunSummaryForTarget, resumePipelinesWithApiKey - Stages (6): identifyComponents, mapComponentsToIds, assignComponentColors, segmentConversation, generateConversationSummary, generateContextAnalysis - Actions (18): reprocess*, apply*, generate*, dimension CRUD, group/delete Pattern: each instrumented function delegates to a _private implementation, with recordCall() wrapping the delegation. This works with ES module immutable bindings (no monkey-patching).
The global callStack broke with concurrent async calls (e.g., two files processed in parallel via Promise.all). Push/pop interleaved across independent async chains, producing wrong parentIndex values. Fix: remove callStack entirely. Instead: - Record startMs/endMs (high-res performance.now() relative to session start) on each entry - Compute parentIndex at session end via time containment: B is a child of A if B's time interval is fully contained within A's, and A is the tightest such container - Sort entries by startMs before output (they arrive in completion order due to async, but consumers want chronological start order)
Store diffing: - Replace storeBefore/storeAfter with storeDiff (compact delta) - Each conversation is compacted to metadata: id, filename, status, step, messageCount, totalParts, dimension summaries (components, colors, mapping count), static component info, stepTimings, warnings - Diff shows only conversations/groups that were added, removed, or changed - Changed entries show [oldValue, newValue] per field - Reduces store data from ~63% of file size to near-zero for unchanged state New instrumentation (9 functions): - identifyForDimension, classifyForDimension, colorForDimension (dimension-level orchestrators — the testable AI stage units) - summarizeConversation, staticComponentise, addTokenCounts (key pure operations called within stages) - parse (stage entry point), parseFileContent, parseFileDropInput (parser chain from file drop to parsed data) - processFileDrop on conversation store (top-level UI entry point) Total instrumented functions: 44
When a wrapper immediately delegates to an inner function within the same millisecond, both entries have identical startMs/endMs. The containment algorithm couldn't distinguish parent from child. Fix: when intervals are identical, the entry with the lower index (called first) is the parent. Also improved tiebreaking: prefer latest start, then earliest end, then lowest index.
When A calls B calls C and all three complete in the same millisecond, all entries have identical startMs/endMs. Previously C picked A as parent (lowest index) instead of B (most immediate caller). Fix: when candidate intervals are identical, prefer the highest index (closest preceding caller) rather than the lowest.
Self-contained instruction document for a coding session that receives session recording JSON files and writes unit tests. Covers: - Session log format (entries, args, results, storeDiff, call tree) - Test patterns for each layer (operations, parsers, stages, pipeline) - How to mock AI SDK calls using recorded results - How to test idempotency in dimension-level orchestrators - How to use storeDiff for side-effect assertions - Step-by-step checklist for reading a recording and writing tests
… segmentation patterns Added from analysis of the extensive 'Compaction Everything' recording (96 entries, 202s, covering file drop + segmentation edit + prompt edits + apply-to-all + grouping): - Reprocessing tests: reprocessTarget with context modifiers, dimNames - applyPromptsToAll: cross-conversation prompt propagation tests - groupConversations: Zustand store group CRUD tests - Segmentation: testing both fast path (no large parts) and AI path - Session type identification from top-level entries - Idempotency detection: compare multiple entries of same function - Updated mock table with actions and store layers - Note on testing actions vs testing the pipeline they delegate to
210 tests across 24 test files covering operations, parsers, stages, pipeline, and stores. Tests use real session recording data from two captured browser sessions as ground truth for assertions. Test infrastructure: - src/__tests__/fixtures/inputs/ — conversation files from recordings - src/__tests__/fixtures/recordings/ — session recording JSON files - src/__tests__/helpers.ts — file loading utilities - src/__tests__/recording-fixtures.ts — extracted AI results as constants Coverage (non-UI): - model/: 96% stmts - operations/: 95% stmts (100% for 5 of 7 files) - stages/: 93% stmts (100% funcs) - stages/ai/: 67% stmts - pipeline/: 72% stmts - parsers/: 64% stmts - stores/: 15% stmts (conversation-store at 77%) Key patterns: - Pure functions tested with exact values from recordings - AI stages mocked with recorded AI responses - Pipeline tests use real segmentation patterns from recordings - Classify mocks build mappings dynamically from actual part IDs - Idempotency tested with both recordings (skip and re-run cases) - Reprocessing workflows from compaction-everything.json tested end-to-end: segmentation change, prompt edits, apply-to-all
Adds .github/workflows/test.yml that runs tests on every PR and push to main. To enforce passing tests before merge, enable branch protection: Repository Settings → Branches → Branch protection rules → Add rule Branch name pattern: main ✓ Require status checks to pass before merging ✓ Require branches to be up to date before merging Status checks: select 'test'
- Copy codex-sample.jsonl and completions-ask_github_gc.json into src/__tests__/fixtures/inputs/ so tests don't depend on uncommitted sample-logs/ directory - Remove swing_stories_input.json test (private data, repo is public) - Remove loadSampleLog helper (no longer used) - All test inputs now live under src/__tests__/fixtures/inputs/
docs/spec.md — 22 sections covering every stage of the pipeline, all pure operations, the store, and supporting infrastructure. Each property is something the test suite enforces.
Member
Author
|
Superseded by #4 (rewrite branch). Closing this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.