feat(video): text layer resize, layer mirroring, pipeline deletion fix, and text overlay rendering fix#97
Closed
staging-devin-ai-integration[bot] wants to merge 135 commits intomainfrom
Closed
feat(video): text layer resize, layer mirroring, pipeline deletion fix, and text overlay rendering fix#97staging-devin-ai-integration[bot] wants to merge 135 commits intomainfrom
staging-devin-ai-integration[bot] wants to merge 135 commits intomainfrom
Conversation
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Add dual input pins: 'audio' (Opus) and 'video' (VP9), both optional - Add video track via VideoCodecId::VP9 with configurable width/height - Multiplex audio and video frames using tokio::select! in receive loop - Track monotonic timestamps across tracks (clamp to last_written_ns) - Convert timestamps from microseconds to nanoseconds for webm crate - Dynamic content-type: video/webm;codecs="vp9,opus" | vp9 | opus - Extract flush logic into flush_output() helper - Add video_width/video_height to WebMMuxerConfig - Add MuxTracks struct and webm_content_type() const helper - Update node registration description - Add test: VP9 video-only encode->mux produces parseable WebM - Add test: no-inputs-connected returns error - Update existing tests to use new 'audio' pin name Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- YAML compiler: add Needs::Map variant for named pin targeting - Color Bars Generator: SMPTE I420 source node (video::colorbars) - MoQ Peer: video input pin, catalog with VP9, track publishing - Frontend: generalize MSEPlayer for audio/video, ConvertView video support - Frontend: MoQ video playback via Hang Video.Renderer in StreamView - Sample pipelines: oneshot (color bars -> VP9 -> WebM) and dynamic (MoQ stream) Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Detect pipelines without http_input as no-input (hides upload UI) - Add checkIfVideoPipeline helper for video pipeline detection - Update output mode label: 'Play Video' for video pipelines - Derive isVideoPipeline from pipeline YAML via useMemo Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…edia-generic UI messages Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…only pipelines - ColorBarsNode now draws a 4px bright-white vertical bar that sweeps across the frame at 4px/frame, making motion clearly visible. - extractMoqPeerSettings returns hasInputBroadcast so the UI can infer whether a pipeline expects a publisher. - handleTemplateSelect auto-sets enablePublish=false for receive-only pipelines (no input_broadcast), skipping microphone access. - decideConnect respects enablePublish in session mode instead of always forcing shouldPublish=true. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…necessary metadata clones - Add Vp9EncoderDeadline enum (realtime/good_quality/best_quality) to Vp9EncoderConfig, defaulting to Realtime instead of the previous hard-coded VPX_DL_BEST_QUALITY. - Store deadline in Vp9Encoder struct and use it in encode_frame/flush. - Encoder input task: use .take() instead of .clone() on frame metadata since the frame is moved into the channel anyway. - Decoder decode_packet: peek ahead and only clone metadata when multiple frames are produced; move it on the last iteration. - Encoder drain_packets: same peek-ahead pattern to avoid cloning metadata on the last (typically only) output packet. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Add verifyVideoPlayback helper for MSEPlayer video element verification - Add verifyCanvasRendering helper for canvas-based video frame verification - Add convert view test: select video colorbars template, generate, verify video player - Add stream view test: create MoQ video session, connect, verify canvas rendering Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…n text in asset mode - mixing.yml: use 'audio' input pin for webm_muxer instead of default 'in' pin - ConvertView: show 'Convert File' button text when in asset mode (not 'Generate') - test-helpers: fix prettier formatting Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…ction Replace fixed 'audio'/'video' pin names with generic 'in'/'in_1' pins that accept both EncodedAudio(Opus) and EncodedVideo(VP9). The actual media type is detected at runtime by inspecting the first packet's content_type field (video/* → video track, everything else → audio). This makes the muxer future-proof for additional track types (subtitles, data channels, etc.) without requiring pin-name changes. Pin layout is config-driven: - Default (no video dimensions): single 'in' pin — fully backward compatible with existing audio-only pipelines. - With video_width/video_height > 0: two pins 'in' + 'in_1'. Updated all affected sample pipelines and documentation. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…input_types Replace packet probing with connection-time media type detection. The graph builder now populates NodeContext.input_types with the upstream output's PacketType for each connected pin, so the webm muxer can classify inputs as audio or video without inspecting any packets. Changes: - Add input_types: HashMap<String, PacketType> to NodeContext - Populate input_types in graph_builder (oneshot pipelines) - Leave empty in dynamic_actor (connections happen after spawn) - Refactor WebMMuxerNode::run() to use input_types instead of probing - Remove first-packet buffering logic from receive loop - Update all NodeContext constructions in test code - Update docs to reflect connection-time detection Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…lays, and spawn_blocking Implements the video::compositor node (PR3 from VIDEO_SUPPORT_PLAN.md): - Dynamic input pins (PinCardinality::Dynamic) for attaching arbitrary raw video inputs at runtime - RGBA8 output canvas with configurable dimensions (default 1280x720) - Image overlays: decoded once at init via the `image` crate (PNG/JPEG) - Text overlays: rasterized once per UpdateParams via `tiny-skia` - Compositing runs in spawn_blocking to avoid blocking the async runtime - Nearest-neighbor scaling for MVP (bilinear/GPU follow-up) - Per-layer opacity and rect positioning - NodeControlMessage::UpdateParams support for live parameter tuning - Pool-based buffer allocation via VideoFramePool - Metadata propagation (timestamp, duration, sequence) from first input New dependencies: - image 0.25.9 (MIT/Apache-2.0) — PNG/JPEG decoding, features: png, jpeg - tiny-skia 0.12.0 (BSD-3-Clause) — 2D rendering, pure Rust - base64 0.22 (MIT/Apache-2.0) — base64 decoding for image overlay data 14 tests covering compositing helpers, config validation, node integration, metadata preservation, and pool usage. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Fix shutdown propagation: add should_stop flag so Shutdown in the non-blocking try_recv loop properly breaks the outer loop instead of falling through to an extra composite pass. - Fix canvas resize: remove stale canvas_w/canvas_h locals captured once at init; read self.config.width/height directly so UpdateParams dimension changes take effect immediately. - Fix image overlay re-decode: always re-decode image overlays on UpdateParams, not only when the count changes (content/rect/opacity changes were silently ignored). - Add video_compositor_demo.yml oneshot sample pipeline: colorbars → compositor (with text overlay) → VP9 → WebM → HTTP output. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
serde_saphyr cannot deserialize YAML with 4+ nesting levels inside params when the top-level type is an untagged enum (UserPipeline). Text/image overlays with nested rect objects trigger this limitation. Removed text_overlays from the static sample YAML. Overlays can still be configured at runtime via UpdateParams (JSON, not serde_saphyr). Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…t pipelines Mirrors the AudioMixerNode pattern: when num_inputs is set in params, pre-create input pins so the graph builder can wire connections at startup. Single input uses pin name 'in' (matching YAML convention), multiple inputs use 'in_0', 'in_1', etc. The sample pipeline now sets num_inputs: 1 so the compositor declares the 'in' pin that the graph builder expects. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Colorbars node: add pixel_format config (i420 default, rgba8 supported) with RGBA8 generation + sweep bar functions - Compositor: accept both I420 and RGBA8 inputs (auto-converts I420 to RGBA8 internally for compositing via BT.601 conversion) - Compositor: add output_pixel_format config (rgba8 default, i420 for VP9 encoder compatibility) with RGBA8→I420 output conversion - Sample pipeline: uses I420 colorbars → compositor (output_pixel_format: i420) → VP9 encoder → WebM muxer → HTTP output Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
The non-blocking try_recv loop was draining all queued frames and keeping only the latest per slot. When spawn_blocking compositing was slower than the producer (colorbars at 90 frames), intermediate frames were dropped, resulting in only 2 output frames. Changed to take at most one frame per slot per loop iteration so every produced frame is composited and forwarded downstream. Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Non-first layers without explicit layers config are auto-positioned as PiP windows (bottom-right corner, 1/3 canvas size, 0.9 opacity) - Sample pipeline now uses two colorbars sources: 640x480 I420 background + 320x240 RGBA8 PiP overlay, making compositing visually obvious Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Previously I420→RGBA8 (input) and RGBA8→I420 (output) conversions ran on the async runtime, blocking it for ~307K pixel iterations per frame per input. Now all conversions run inside the spawn_blocking task alongside compositing, keeping the async runtime free for channel ops. - Removed ensure_rgba8() calls from frame receive paths - Store raw frames (I420 or RGBA8) in InputSlot.latest_frame - Added pixel_format field to LayerSnapshot - composite_frame() converts I420→RGBA8 on-the-fly per layer - RGBA8→I420 output conversion also runs inside spawn_blocking Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
…, and selectability (#78) * fix(compositor): improve image overlay quality, caching, aspect ratio, and selectability - Replace nearest-neighbor prescaling with bilinear (image crate Triangle filter) for much better rendering of images containing text or fine detail - Cache decoded image overlays across UpdateParams calls — only re-decode when data_base64 or target rect dimensions change, reusing existing Arc<DecodedOverlay> otherwise - Lock aspect ratio for image layers during resize (same as video layers) - Show actual image thumbnail in compositor canvas UI for easier selection; switch border from dotted to solid, remove crosshatch pattern Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): guard against index mismatch in image overlay cache Use old_imgs.get(i) instead of old_imgs[i] to avoid a panic when a previous decode_image_overlay call failed, leaving old_imgs shorter than old_cfgs. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): address review — proper index mapping for cache, broader MIME detection - Build a HashMap<usize, &Arc<DecodedOverlay>> by walking old configs and decoded overlays in tandem, so cache lookups use config index rather than assuming positional alignment (which breaks when a previous decode failed) - Add WebP and GIF magic-byte detection for image thumbnail data URIs Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style(compositor): apply cargo fmt formatting Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): fix HashMap type and double-deref in overlay cache Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): content-keyed overlay cache with dimension-based matching Replace incorrect positional index mapping with a content-keyed cache that matches decoded overlays to configs by comparing prescaled bitmap dimensions against the config's target rect. This correctly handles the case where a mid-list decode failure makes the decoded slice shorter than the config vec — failed configs are skipped (not consumed) because their target dimensions won't match the next decoded overlay. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): default image overlay z-index to 200 so it renders above video layers Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style(compositor): add rationale comment for clippy::expect_used suppression Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: apply formatting fixes (cargo fmt + prettier) Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): address review findings #1-#4, #7 - Replace dimension-matching cache heuristic with index-based mapping using image_overlay_cfg_indices (finding #1) - Only update x/y position on cache hit, not full rect clone (finding #2) - Fix MIME sniffing comment wording to 'base64-encoded magic bytes', add BMP detection (finding #3) - Switch from data-URI to URL.createObjectURL with cleanup for image overlay thumbnails (finding #4) - Change SAFETY comment to Invariant in prescale_rgba (finding #7) Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): preserve image aspect ratio, add image layer controls, optimize base64 decode - Backend: prescale images with aspect-ratio preservation (scale-to-fit instead of stretch-to-fill) and centre within the target rect. - Backend: re-centre cached overlays on position update. - Frontend: detect natural image dimensions on add and set initial rect to match source aspect ratio. - Frontend: add opacity/rotation slider controls for selected image overlays (matching video and text layer controls). - Frontend: fix findAnyLayer to pass through rotationDegrees and zIndex for image overlays instead of hardcoding 0. - Frontend: replace O(n) atob + byte-by-byte loop with fetch(data-URI) for more efficient base64-to-blob conversion. - Frontend: remove BMP MIME detection (inconsistent browser support). - Frontend: add z-index band allocation comments (video 0-99, text 100-199, image 200+). Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): apply rotation transform to image overlay layer in canvas preview Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(compositor): include rotationDegrees and zIndex in overlay sync change detection Add rotationDegrees and zIndex to the image overlay change-detection comparisons in the params sync effect so that YAML or backend changes to these fields are reflected in the UI. Also add the missing zIndex check to the text overlay change detection for consistency. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
) * refactor: consolidate compositor layer UI controls and hook helpers - Remove dead OverlayList component (superseded by UnifiedLayerList) - Remove unused OverlaySection, OverlaySectionHeader, OverlayItem styled components - Extract LayerPropertyControls component for shared header, opacity slider, and rotation slider across all layer types (video, text, image) - Refactor UnifiedLayerList to use LayerPropertyControls, eliminating ~130 lines of triplicated slider JSX - Extract mergeOverlayState<T> generic helper for sync effect, replacing duplicated merge+diff logic for video, text, and image layers - Extract updateOverlay/removeOverlay generic helpers in the hook, consolidating the duplicated update/remove callbacks for text and image overlays Signed-off-by: Devin AI <devin@cognition.ai> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: preserve video layer control order (Header → Opacity → Rotation → Z-index) Move z-index controls from children to footer prop in LayerPropertyControls so video layers render controls in the original order. Add footer slot to LayerPropertyControls for content rendered after the shared sliders. Signed-off-by: Devin AI <devin@cognition.ai> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: unified z-index controls and auto-select on add - Replace per-type z-index bands with actual zIndex values in entries - Add unified stack navigation (sortedAllByZ) across all layer types - Extract renderZIndexFooter helper and use it for video, text, and image - Auto-select newly added text/image overlays via setSelectedLayerId - Default text overlay zIndex to 100+count instead of 0 Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: default parsed text overlay zIndex to 100+i Aligns the parse-time default with the add-time default so existing text overlays without a persisted z_index land in the correct band (100-199) instead of at 0 (below video layers). Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: use actual zIndex in findAnyLayer for text overlays Was hardcoded to 0 instead of textOverlay.zIndex, inconsistent with the image overlay branch which correctly used imgOverlay.zIndex. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…ing fix (#80) * feat(compositor): text color, font selection, draggable layers, clipping fix - Text color: add color picker (RGB + alpha slider) for text overlays - Font selection: add font_name field to TextOverlayConfig with 12 curated system fonts (DejaVu, Liberation, FreeFonts), dropdown in UI, warning when named font file is missing - Draggable layer list: replace z-index ▲/▼ buttons with drag-to-reorder using motion/react Reorder, reassigns z-indices on drop - Text clipping fix: expand bitmap height to ceil(font_size * 1.4) in rasterize_text_overlay and auto-expand UI rect height when font size increases Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: apply rustfmt to overlay.rs Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: batch reorder z-index updates and add color comparison - Add reorderLayers() to useCompositorLayers that atomically updates z-index for all layer types (video, text, image) in a single commit, preventing race conditions from stale refs when handleReorder fired N individual update calls. - Add missing color array comparison to mergeOverlayState's hasExtraChanges comparator so server-echoed color changes are correctly detected. - Remove unused onZIndexChange prop from UnifiedLayerList since reorderLayers now handles all z-index mutations. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: use dynamic max z-index for new overlays instead of fixed bands Replace fixed z-index bands (text: 100+, image: 200+) with maxZIndex() + 1 so that new overlays always stack on top even after drag-to-reorder has normalized z-indices to [0, n-1]. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: expand text overlay bitmap to measured text dimensions Add measure_text() that computes actual pixel width/height from font metrics, and use it in rasterize_text_overlay to expand the bitmap to fit the full rendered string. Previously only height was expanded via a 1.4× heuristic; now both width and height use exact font measurements. On the UI side, updateTextOverlay now auto-expands the rect width (~0.6× font_size per character) in addition to height when the text would overflow the current box. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: cargo fmt Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: address review — atomic reorder commit and complete font list - Replace throttledConfigChange with immediate onParamChange in reorderLayers' onParamChange path so video layer and overlay z-index updates commit atomically in the same tick. - Extract serializeLayers() helper to avoid duplicating layer serialization logic between buildConfig and reorderLayers. - Add missing dejavu-serif-bold and dejavu-sans-mono-bold to the font_name doc comment in TextOverlayConfig. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * feat: render font selection in canvas preview and fix dropdown contrast - Add FONT_FAMILY_MAP to CompositorCanvas that maps backend font names to CSS font-family values. Text overlays now preview the selected font in both display and edit mode on the canvas. - Bold font variants (e.g. dejavu-sans-bold) render with fontWeight 700. - Fix FontSelect contrast: use var(--sk-panel-bg) instead of undefined var(--sk-input-bg), add color-scheme hint, and style option elements explicitly for dark mode compatibility. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: improve font preview fallbacks and remove alpha slider - Add web-safe intermediate fonts (Verdana, Georgia, Arial, Times New Roman, Courier New) to the CSS font-family fallback stacks so the canvas preview shows a visible difference between sans-serif, serif, and monospace font groups even when the exact system fonts are not installed in the browser. - Remove the alpha slider from text color controls. Text opacity is already covered by the layer opacity slider, and a standalone alpha slider for a single channel was confusing. The color picker now always sends alpha=255. Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: preserve existing alpha when changing text color Signed-off-by: Devin AI <devin@devin.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Signed-off-by: Devin AI <devin@devin.ai> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Extract shared view layout primitives, live indicator badge, stream callbacks, and video canvas hook to reduce code duplication across StreamView, ConvertView, MonitorView, and node components. - Extract 11 shared styled components to components/ui/ViewLayout.ts - Extract LiveDot/LiveBadge to components/ui/LiveIndicator.ts - Consolidate identical stream callbacks via makeStreamCallbacks factory - Extract useVideoCanvas hook from StreamView and OutputPreviewPanel Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* perf(compositor): SIMD alpha scan, font cache, fat LTO Three profiling-driven optimizations: 1. first_layer_all_opaque: replace the scalar chunks_exact(4).all() scan with the existing AVX2/SSE2 all_alpha_opaque helpers. The Arc::as_ptr cache key changes every frame for video sources, so the scan runs on the full RGBA buffer per frame (~19.7% of total time in profiling). Add a safe all_alpha_opaque() wrapper to pixel_ops/mod.rs that encapsulates the cfg + feature detection + unsafe dispatch so kernel.rs stays unsafe-free. 2. Font re-parsing: cache parsed fontdue::Font objects in a process- wide LazyLock<Mutex<HashMap>> keyed by the resolved font source (filesystem path, or hash of inline base64 data). load_font now returns Arc<Font>; repeated calls for the same font are an Arc::clone instead of a fresh fs::read + from_bytes parse. Splits the old load_font into resolve_font_source (cheap: builds the key + a lazy byte loader) and load_font (check cache, else invoke loader + parse + insert). File I/O and base64 decode are skipped entirely on cache hits; the lock is never held across the parse. Eliminates the ~3.5s Font::from_bytes cost observed when overlay params update frequently. 3. LTO: switch profile.release from thin to fat LTO. Profiling showed core::ub_checks::maybe_is_nonoverlapping at 3.75s flat despite codegen-units = 1; fat LTO gives LLVM full cross-crate visibility to eliminate these precondition checks. Trade-off is longer release build times. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * more optimizations --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat(video): bundle DejaVu fonts at compile time Embed all 6 DejaVu font variants (Sans, Sans Bold, Sans Mono, Sans Mono the dependency on system-installed font packages. - Add new crates/nodes/src/video/fonts.rs module with compile-time embedded font data and lookup helpers - Replace system-path-based KNOWN_FONTS map in overlay.rs with bundled font lookups (no filesystem I/O for named fonts) - Update colorbars draw_time to use embedded DejaVu Sans Mono as primary font, with optional font_path override for custom fonts - Remove Liberation and FreeFonts from named font set (GPL licensing) - Update UI font options to match the bundled DejaVu-only set - Update sample pipeline to use font_name instead of font_path - Add REUSE.toml annotations for bundled font files (Bitstream Vera license) The font_path and font_data_base64 config fields remain for loading external/custom fonts from the filesystem or inline data. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: apply cargo fmt formatting Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: gate fonts module behind colorbars/compositor features Avoids compiling ~2.8 MB of embedded font data into non-video builds. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(video): gracefully fall back for unknown font_name values Instead of returning an error (which produces placeholder rectangles), unknown font names now log a warning and fall back to the bundled DejaVu Sans default. This handles legacy font names (e.g. liberation-*, free*) that were removed from the bundled set. Also removes the unnecessary Result wrapper from resolve_font_source since it can no longer fail. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* fix: address video branch review findings - Move lto=fat and codegen-units=1 to a dedicated release-lto profile so CI uses the faster default release profile - Remove unused _scratch parameter from convert_rgba8_to_nv12_frame - Change VideoFrame constructors from panic to Result<Self, StreamKitError> for layout mismatches and undersized buffers - Regenerate api-types.ts to include Nv12 in PixelFormat union Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: cargo fmt Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat(video): add webcam input support and PiP demo
Add support for publishing webcam video from the browser alongside the
existing microphone audio. The hang library's Camera source captures
video and publishes it over MoQ, where the server-side moq_peer node
receives it and forwards it into the processing pipeline.
Backend changes (peer.rs):
- Add 'video_out' output pin to MoqPeerNode for forwarding publisher
video into the pipeline
- Generalize wait_for_catalog_with_audio into wait_for_catalog that
discovers both audio and video tracks from the publisher's catalog
- Add process_publisher_tracks to concurrently receive from both audio
and video track consumers, routing audio to 'out' and video to
'video_out'
Frontend changes:
- Add Hang.Publish.Source.Camera to the publish path alongside
Microphone, wiring it as the video source for the broadcast
- Add camera toggle button and status display in StreamView
- Add camera state, cleanup, and toggleCamera action to streamStore
New sample pipeline (video_moq_webcam_pip.yml):
- Composites user's webcam as PiP over colorbars background
- Includes a 'Hello from StreamKit' text overlay
- Full roundtrip: webcam -> VP9 decode -> compositor -> VP9 encode ->
subscribers
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: format peer.rs with cargo fmt
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(video): add camera permission timeout and camera to setMoqRefs
- Add 10-second camera permission timeout in schedulePostConnectWarnings
so cameraStatus transitions to 'error' if camera is never acquired,
matching the existing microphone timeout behavior.
- Add camera field to setMoqRefs type signature and implementation.
- Update test mock to include camera in setMoqRefs call.
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(video): handle incremental catalog updates for webcam
The hang library publishes catalog updates incrementally as browser
permissions are granted (mic first, then camera). The previous
wait_for_catalog returned on the first catalog with any track, missing
the video track that arrives in a later catalog update.
Replace the one-shot wait_for_catalog + process_publisher_tracks pattern
with watch_catalog_and_process that continuously monitors the catalog
and spawns a processing task for each track as it appears. This handles
the common case where mic and camera permissions are granted at
different times.
Also extract spawn_track_processor and await_track_tasks helpers to
keep cognitive complexity within clippy limits.
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(video): enable HD video encoder for camera publishing
The Publish.Video.Props type has source, hd, sd, and flip — but the hd
and sd encoders default to disabled. Without explicitly enabling at
least one encoder, the camera source is acquired but no video frames
are encoded, so the catalog never contains video renditions.
Add hd: { enabled: true } to enable the HD video encoder.
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(video): add diagnostic logging and concurrent task awaiting
- Add tracing to spawn_track_processor start/finish
- Add output_pin context to get_next_group and process_frame_from_group logs
- Rewrite await_track_tasks to use tokio::select! so video task early
exits are immediately visible (previously blocked behind audio task)
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: format peer.rs with cargo fmt
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(video): prevent await_track_tasks from hanging with single track
Initialize audio_done/video_done based on whether the corresponding
handle exists, so the while loop terminates when only one track type
is present (e.g. audio-only publisher).
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): patch @moq/hang encoder effect-scoping bug
The webcam PiP overlay never rendered because the video/hd subscription
was cancelled ~4ms after it started, during the camera's permission-grant
→ device-enumeration cascade.
Root cause is a lexical-scoping bug in @moq/hang@0.1.2's
publish/video/encoder.js. Inside Encoder#serve, the config-watching child
effect was declared as `effect.effect(() => {...})` without an (effect)
parameter, so `effect.get(this.#config)` inside it resolved `effect` to the
outer closure variable — the request-handling effect from broadcast.js.
That subscribed the *parent* effect to #config. When camera.source flapped
to undefined (Camera#run re-runs after device.requested changes, and its
effect.set cleanup resets the signal), #config followed, and the parent
re-ran — tearing down the MoQ track producer (server sees 'subscribe
cancelled') and the VideoEncoder. The parent's next body run saw
request.track.state.closed === true and returned, leaving the encoder
permanently dead with no re-subscription path.
The same bug persists in @moq/publish@0.2.2, so upgrading hang to 0.2.0
(which is a breaking package split) would not help.
The patch gives the child effect its own (effect) parameter so only the
3-line configure body re-runs on #config flaps — the producer, encoder,
and MoQ track all survive. Also guards encoder.close() against the 'closed'
state to silence the InvalidStateError console noise on teardown.
Applied via `bun patch`; patchedDependencies in package.json makes bun
re-apply it on every install. REUSE.toml annotates ui/patches/** with the
upstream (MIT OR Apache-2.0) license since the diff context is verbatim
from @moq/hang.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(moq): re-subscribe on transient publisher track cancellation
Hardening independent of the @moq/hang patch. Previously, if the browser
transiently cancelled the video/hd track (e.g. the camera.source flap in
unpatched @moq/hang), spawn_track_processor's task would error out on
moq_lite::Error::Cancel from next_group() and never recover — and
watch_catalog_and_process had already stopped watching the catalog after
discovering both tracks, so there was no re-subscribe path.
This moves subscribe_track() inside the spawned task and wraps it in a
bounded retry loop (3 attempts, 100ms backoff, yields to shutdown during
backoff). BroadcastConsumer is Clone (Arc state + watch::Receiver +
async_channel::Sender), so cloning it into the task is cheap and lets us
re-subscribe after cancellation. moq-lite evicts the cancelled track
producer from its dedup map once unused, so the re-subscribe creates a
fresh TrackProducer/Consumer pair, and the browser's Broadcast#runBroadcast
for(;;) loop happily serves the new request.
To preserve the typed error through the call chain, get_next_group now
returns the raw moq_lite::Error and process_publisher_frames classifies
Error::Cancel into a new TrackExit::Cancelled outcome. The JoinHandle
return type is unchanged, so await_track_tasks needs no modification.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* chore(test): add panics doc to create_test_video_frame
Pre-existing lint failure on the branch: just lint-skit's second clippy
invocation (`--workspace --exclude streamkit-server --all-targets`) builds
streamkit-nodes' lib-test target and trips clippy::unwrap_used +
clippy::missing_panics_doc on line 159, introduced in af1f4de (#84). It
passed before only because of incremental-compilation caching.
Follows the existing convention in this file (see assert_state_update at
line ~185): use .expect() with a descriptive message, document the panic
in a # Panics section, and #[allow(clippy::expect_used)] since this is a
cfg(test)-gated test helper where a panic on bad test inputs is the
intended behaviour.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fixes
* review feedback
* rework compositor
* review
* doc updates
---------
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…#86) - compositor/mod.rs: use f64::from() for u32->f64 widening in fit_rect_preserving_aspect (clippy::cast_lossless, 6 sites) - compositor/mod.rs: drop redundant stop_reason assignment in Shutdown arm; the initializer at declaration already carries "shutdown" (rustc unused_assignments — all break paths now assign since aa0347b) - peer.rs: drop u128->u64 cast in tracing field; tracing accepts u128 directly (clippy::cast_possible_truncation, matches pull.rs:780) - LICENSES/Bitstream-Vera.txt: add SPDX license text referenced by REUSE.toml for bundled DejaVu fonts (reuse lint) Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat: upgrade moq ecosystem dependencies to latest versions Backend: - hang 0.13.0 → 0.15.1 - moq-lite 0.13.0 → 0.15.0 - moq-native 0.12.1 → 0.13.2 - Remove dead moq-transport dependency API migrations: - Request.accept() → Request.ok() - Request.reject() → Request.close() - Frame.keyframe field removed; use OrderedProducer::keyframe() - moq_lite::coding module now private; use hang::container::Timestamp - BroadcastConsumer::subscribe_track() now returns Result - BroadcastProducer::create_track() now returns Result - TrackProducer::write_frame() now returns Result - TrackProducer::close() → finish() - catalog_to_json shim removed; use Catalog::to_string() Frontend: - @moq/hang ^0.1.2 → ^0.2.0 (now data-layer only) - Add @moq/watch ^0.2.2 (new Watch.Broadcast decomposition) - Add @moq/publish ^0.2.2 (new Publish.Broadcast) - Broadcast path prop renamed to name - Watch pipeline: manual Sync → Source → Decoder → Emitter/Renderer - Patches moved from @moq/hang to @moq/publish (3 bugs still present) Signed-off-by: Devin AI <devin@cognition.ai> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * chore: remove @moq/publish patch (no longer needed) Signed-off-by: Devin AI <devin@cognition.ai> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: Devin AI <devin@cognition.ai> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…88) * feat(stream): conditional device init based on pipeline media types - extractMoqPeerSettings now detects needsAudioInput/needsVideoInput by scanning which downstream nodes consume from the moq_peer's output pins - streamStore only creates Microphone/Camera sources for the media types the pipeline actually needs, avoiding unnecessary permission prompts - StreamView conditionally shows mic/camera toggle buttons and status text based on the pipeline's media requirements - Add audio path (opus decode → gain → opus encode) to the webcam PiP pipeline so it handles both audio and video Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(pipeline): use correct input pin name 'in' for moq_peer audio The moq_peer node's audio input pin is named 'in' (not 'audio'). The previous needs map key 'audio' was silently unconnected. Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * test: add playwright e2e test for webcam PiP pipeline Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * feat(moq_peer): type-agnostic pins with runtime media kind inference - Input pins (in, in_1) now accept both EncodedAudio(Opus) and EncodedVideo(VP9) instead of being hardcoded to a single type - Output pins (out, out_1) produce PacketType::Any - Media kind is inferred at runtime from NodeContext::input_types - Symmetric pin mapping: in ↔ out, in_1 ↔ out_1 - Renamed video → in_1, video_out → out_1 for type-neutral naming - Updated all pipeline YAMLs and UI references accordingly - Added camera permission to playwright config Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(moq_peer): handle empty input_types in dynamic pipelines Dynamic pipelines don't populate NodeContext::input_types, so pin kinds were always None — causing packets to be silently dropped and the subscriber catalog to advertise no tracks. Fix: - Add infer_kind_from_packet() to classify packets by content_type field at runtime (VP9 sets "video/vp9", Opus leaves it None) - Lazily determine pin kind on first packet when input_types is empty - Optimistically advertise both audio and video tracks in the subscriber catalog when pin types are unknown (unused tracks stay idle) - Default output pin mapping: audio → "out", video → "out_1" when types are not known at startup Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: conditional watch setup and lazy media kind inference - peer.rs: Remove optimistic has_audio/has_video defaults for dynamic pipelines. Instead, update lazily when first packet arrives on each pin. This fixes colorbars (video-only) incorrectly advertising audio. - moqPeerSettings.ts: Add detectPeerOutputMediaTypes() that determines what media types a pipeline outputs to subscribers by checking the kinds of nodes connected to moq_peer's input pins. - streamStore.ts: Add pipelineOutputsAudio/pipelineOutputsVideo flags. Make setupWatchPath conditional — only create video renderer when the pipeline outputs video, and audio emitter when it outputs audio. - StreamView.tsx: Set output type flags when selecting templates. Reset to (true, true) in direct mode. - MonitorView.tsx: Detect output types from pipeline connections when starting preview. Only create relevant watch-side components. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(moq_peer): use watch channel to resolve subscriber catalog race condition For dynamic pipelines, has_audio/has_video start as false and are updated lazily on first packet arrival. The subscriber catalog was built using a snapshot of these values at connection time, causing an empty catalog if the subscriber connected before the first pipeline packet arrived. Now a tokio::sync::watch channel communicates the resolved media type state from the main select loop to subscriber tasks. Subscribers wait (up to 5s) for all connected pin types to be determined before building the MoQ catalog, guaranteeing the catalog accurately reflects the media types that actually flow through the pipeline. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(e2e): scroll canvas into view for video rendering tests The Watch.Video.Renderer uses IntersectionObserver to enable the Video.Decoder (which subscribes to the video/data MoQ track). In headless Chromium, the canvas element was below the viewport fold, so IntersectionObserver never reported it as intersecting. This prevented the decoder from subscribing to the video track, resulting in a black canvas. Fix: scroll the canvas into view before the wait period so the IntersectionObserver fires and the video decoder subscribes to the video/data track. Also keeps the optimistic catalog approach for dynamic pipelines where has_audio=true and has_video=true are set immediately so the catalog track exists before the browser subscribes. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: apply cargo fmt Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * docs(AGENTS): add e2e testing guidelines and headless-browser pitfalls Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(moq_peer): correct output pin mapping and add content_type safeguard - Fix audio_output_pin collision when pin_0 carries Video (both pins would resolve to "out"). Add explicit arm for (Some(Video), _) case. - Add debug_assert and trace log in infer_kind_from_packet to catch future encoders that omit content_type on video packets. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): guard toggleMicrophone against missing audio in publish path When pipelineNeedsAudio is false, the publish broadcast is created without an audio property. Guard the store action with optional chaining so it doesn't crash if called programmatically on a video-only pipeline. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style(moq_peer): add rationale comments to clippy allow attributes Per AGENTS.md linting discipline rule, #[allow(...)] exceptions must include explanatory comments. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: Devin AI <devin@cognition.ai> Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…r interactions (#89) * perf(ui): stabilize compositor callbacks with paramsRef Replace direct params usage in throttledConfigChange, throttledOverlayCommit, commitOverlays, and reorderLayers closures with a paramsRef that tracks the latest value at call-time. This eliminates ~12 cascading callback recreations per params reference change (server echo-back after config update) and prevents throttle function recreation which drops pending trailing calls. Add regression test verifying callback referential stability across params changes. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): batch nodeParamsStore updates and memoize layer refs - Add setParams() to nodeParamsStore for batch multi-key updates in a single store mutation (was N separate setParam calls → N intermediate states, N selector re-evaluations for all subscribers). - Use setParams in tuneNodeConfig, handleNodeParamsChanged, and updateStagedNodeParams to reduce store churn during compositor config changes (opacity, rotation, etc.). - Memoize setLayerRef callbacks in CompositorCanvas per layer id so React.memo on VideoLayer/TextOverlayLayer/ImageOverlayLayer is not defeated by new function references on every render. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): stabilize entries + memoize callbacks in UnifiedLayerList - Add structural equality check on entries array via prevEntriesRef to prevent Reorder.Group context invalidation during opacity/rotation slider drags. Entries only track id/kind/zIndex/visible so they don't change during those interactions, but the useMemo produced a new array reference each time. - Memoize handleSelectedOpacityChange and handleSelectedRotationChange with useCallback keyed to selectedLayerId, replacing inline lambdas that created new function references on every render and defeated React.memo on LayerPropertyControls. Expected improvement: eliminates ~35ms Reorder.Group context cascade (3 ReorderItems × deep tooltip subtrees) per compositor commit during slider interactions. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): extract LayerReorderSection memo boundary to block cascade The Reorder.Group and its items (3 ReorderItemComponent, each with tooltips, icons, buttons = ~50 components per item) were re-rendering on every opacity/ rotation slider tick because UnifiedLayerList re-renders when 'layers' prop changes. Extract the Reorder.Group + items into a separate LayerReorderSection component wrapped in React.memo. This component only receives: - entries (stabilised with prevEntriesRef, same reference during opacity drags) - selectedLayerId (stable during a drag on the same layer) - stable callbacks When entries and selectedLayerId are unchanged, React.memo bails out and the entire 150-component Reorder subtree is skipped. Expected improvement: eliminates ~140 of 150 fiber re-renders per commit during opacity/rotation slider interactions (from profiling data showing 48ms avg → expected ~5-10ms). Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): move entries + property controls out of UnifiedLayerList UnifiedLayerList was re-rendering 107 times during opacity slider drags because it received 'layers' as a prop (which changes on every tick). Even though the Reorder section was already memoised, the component body and its non-Reorder children (Add button, labels, inputs) all re-rendered. Move the entries computation into a useStableEntries hook called from CompositorNode. Move all LayerPropertyControls rendering to CompositorNode. UnifiedLayerList now receives only stable props (entries, selectedLayerId, callbacks) so React.memo bails out entirely during opacity/rotation drags. Expected: UnifiedLayerList + all children (Add button, labels, etc.) skip re-rendering completely. Only CompositorCanvas (visual update) and LayerPropertyControls (slider value) re-render. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* fix(video): text wrapping, canvas cleanup, and YAML debounce - Support explicit newlines and word-wrapping in compositor text overlays by adding wrap_text_lines(), measure_text_wrapped(), and blit_text_wrapped() helpers. Update rasterize_text_overlay() to use the wrapped variants when the overlay rect has a non-zero width. - Clear the canvas in useVideoCanvas when the renderer is removed (e.g. pipeline destroyed) so stale frames don't linger in monitor view. Reset intrinsic dimensions so the aspect-ratio hook also resets. - Debounce the YAML editor's onChange callback (500 ms) in YamlPane so intermediate invalid YAML while typing doesn't trigger a flood of parse-error toasts. CodeMirror manages its own buffer so the editor stays responsive. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): allow text overlay content to overflow the config rect The TextContent container used overflow:hidden which clipped wrapped or multi-line text. Switch to overflow:visible and align-items:flex-start so the text extends beyond the dashed rect, matching the backend compositor which auto-expands the overlay bitmap to fit. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): auto-expand text overlay box height for wrapped text Use a hidden measurement span to read the natural text height and expand the overlay box when wrapped/multi-line text exceeds the configured rect height. This matches the backend compositor which auto-expands the overlay bitmap in the same way. Reverts the previous overflow:visible approach in favour of proper box expansion with overflow:hidden preserved. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): preserve expanded text overlay height during drag applyVisualUpdate was overwriting the DOM element height with the config rect height on every pointer-move frame, reverting the component-level auto-expansion for wrapped text. Now only update width/height when the interaction is a resize, not a pure drag. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): reset aspect ratio when canvas dimensions are zeroed useCanvasAspectRatio only updated the ratio for positive dimensions, leaving a stale value when the renderer was removed and the canvas was reset to 0×0. Now explicitly sets ratio to undefined when dimensions are non-positive. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): add editing to useLayoutEffect deps for text height measurement Without editing in the dependency array, cancelling an edit via Escape while the scale has changed leaves displayHeight collapsed to overlay.height because the effect doesn't re-fire when the measurement span reappears. Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style(ui): format CompositorCanvas.tsx with Prettier Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat(ui): improve compositor node UX
- Add side inspector panel for layer properties (appears right of node)
- Add rotation preset buttons (0, 90, 180, 270) with reset button
- Migrate sliders from native <input range> to Radix Slider components
- Improve default layer labels (Input 0, Text 1, Image 0)
- Increase node width from 280 to 320
- Move visibility toggle left, remove button right (hidden until hover)
- Add outside-click handler to close Add menu
- Add useStableEntries hook for structural stability during drags
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): use 0-360 rotation slider range to match presets
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): cap rotation slider max at 359 to prevent snap-back
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): replace Emotion component selector with CSS class to fix crash
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): widen compositor inspector panel from 200px to 280px
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* refactor(ui): move layer list into side panel and fix re-render regression
- Side panel is now always visible, containing both the layer list
and inspector controls (with a divider between them)
- Node body only contains the canvas section
- Memoized all opacity/rotation callbacks with useCallback to prevent
React.memo on LayerInspector from being defeated during slider drags
- Memoized text overlay children JSX with useMemo so the children prop
doesn't create new references on every render
- Unified opacity/rotation handlers dispatch to the correct update
function (video/text/image) based on selected layer type
- Removed unused LayerControls and hasSelection variables
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style(ui): format CompositorNode.tsx
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ui): use selectedLayerKind to stabilize opacity/rotation callbacks
Replace layers/textOverlays array deps in handleSelectedOpacityChange and
handleSelectedRotationChange with a selectedLayerKind useMemo that returns
a primitive string ('video'|'text'|'image'|null). Since selectedLayerKind
only changes on selection change (not on every slider tick), the callbacks
now stay referentially stable during drags, allowing React.memo on
LayerInspector to bail out correctly.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* perf(ui): split LayerInspector into memoized sub-sections
Replace the monolithic LayerInspector component with three individually
memoized sub-components: InspectorHeaderSection, OpacityControl, and
RotationControl. During an opacity slider drag, only OpacityControl
re-renders. During a rotation drag, only RotationControl re-renders.
The header, text overlay children, preset buttons, tooltips, and the
inactive slider all bail out via React.memo.
This eliminates ~50% of re-render work during slider drags by preventing
the inactive section (and all its Radix Slider internals, SKTooltip,
preset buttons) from re-rendering when its value hasn't changed.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* perf(ui): memoize canvas header to prevent SKTooltip re-renders
The LIVE badge SKTooltip and its Tooltip/Popper/PopperProvider chain
were re-rendering 86 times during slider drags (~458ms total) despite
nothing changing. Wrap the canvas header JSX in useMemo keyed on
showLiveIndicator, canvasWidth and canvasHeight — all stable during
opacity/rotation drags.
Combined with the sub-component split from the previous commit, the
only components that now re-render during slider drags are:
- The active slider control (OpacityControl or RotationControl)
- CompositorCanvas + VideoLayer (expected: visual preview must update)
- CompositorNode itself (unavoidable: hook state changes)
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…P9 encoder (#93) * feat(video): add pixel_convert node and remove RGBA conversion from VP9 encoder - Create dedicated video::pixel_convert processor node for pixel format conversion between RGBA8, NV12, and I420 - Runs CPU-heavy conversion on a persistent spawn_blocking thread - Caches output when input Arc<PooledVideoData> pointer hasn't changed (zero-cost passthrough for static scenes) - Emits OpenTelemetry metrics: frames_converted, frames_passthrough, conversion_duration - Remove inline RGBA→NV12 conversion from VP9 encoder - VP9 encoder now rejects RGBA8 input with a clear error message - Update 3 pipeline YAMLs to insert pixel_convert between compositor and VP9 encoder - Add unit tests: passthrough, caching, conversion, roundtrip, unsupported pair rejection, invalid config Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(bench): insert pixel_convert node in compositor benchmark pipeline The benchmark connects compositor (RGBA8 output) directly to the VP9 encoder, which no longer accepts RGBA8. Insert a pixel_convert node between them, matching the pattern used in the sample pipeline YAMLs. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * perf(bench): add pixel_convert microbenchmark for raw conversion paths Measures throughput for all supported conversion paths (RGBA8↔NV12, RGBA8↔I420) across multiple resolutions. Follows the same pattern as compositor_only benchmark with arg parsing, multi-resolution support, per-iteration reporting, and machine-readable JSON output. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
) * feat(ui): add two-layer render-performance profiling infrastructure Layer 1 — Vitest-native render regression testing (component-level, CI): - measureRenders() utility inspired by Reassure (React.Profiler + stats) - Baseline comparison with JSON snapshot I/O and report formatting - Render-perf tests for useCompositorLayers (opacity, rotation, callback stability) Layer 2 — Playwright + React.Profiler harness (interaction-level profiling): - Dev-only perfOnRender callback exposing window.__PERF_DATA__ - window.__PERF_RESET__() for resetting between Playwright scenarios - CompositorNode wrapped in <React.Profiler> in dev builds - Playwright helpers: resetPerfData, capturePerfData, assertRenderBudget, compareSnapshots Infrastructure: - justfile: perf-ui recipe - package.json: test:perf script Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): correct vitest filter pattern in test:perf script Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * feat(ui): add cascade re-render regression tests and perf baseline - Rewrite render-perf tests to use measureHookRenders() with render count assertions focused on cascade re-render detection - Add initial perf-baselines.json snapshot (opacity/rotation slider drags, mixed updates, callback stability) - Document render performance profiling in AGENTS.md (Layer 1/2, cascade detection pattern, when to use) - Fix duplicate import and extract createOnRender helper in measure.ts Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix(ui): address review — correct AGENTS.md names, gate baseline write - Fix AGENTS.md: collectPerfData/clearPerfData → capturePerfData/resetPerfData - Gate writeBaseline() behind UPDATE_PERF_BASELINE=1 env var so regular test-ui runs never silently overwrite the committed baseline - test:perf script sets the env var automatically Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * test(e2e): add compositor slider perf test measuring cascade re-render prevention - New Playwright test: compositor-perf.spec.ts - Starts Webcam PiP (MoQ Stream) pipeline from stream view - Navigates to monitor view where full compositor node graph renders - Selects each layer and drags opacity/rotation sliders - Captures render metrics via window.__PERF_DATA__ - Asserts render budgets for CompositorNode (max 350 renders) - Detects cascade regressions: sibling components must not exceed 60% of CompositorNode's render count during slider interactions - Fix SPDX headers: replace (c) with © in all 8 new files to match CONTRIBUTING.md convention Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * test(e2e): fix compositor perf test — API session creation, correct selectors, baseline Major changes: - Replace stream-view session creation with API-based approach, eliminating the MoQ WebTransport connection requirement that caused skips in headless CI - Fix layer selectors: use text-based getByText('Text 0') instead of li/[class*='LayerListItem'] which matched 0 elements (actual DOM uses plain div elements) - Fix slider section locators: use filter({hasText: /^Opacity/}) with filter({has: getByRole('slider')}) instead of text=Opacity parent traversal - Add embedded pipeline YAML so test is self-contained - Add dev-mode profiler availability check (graceful skip when not on dev server) - Set render budget to 500 based on observed baseline of ~385 renders (3 layers × 2 sliders × 2 directions × 20 steps + mount renders) - Remove installAudioContextTracker (not needed for API-based flow) Verified: all 3 layers found, all 6 sliders exercised, 385 renders measured. Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * chore: add just perf-e2e target, fix baseline overwrite order, update AGENTS.md Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: use bunx and add install-e2e dep for perf-e2e target Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
…nel (#95) * feat: server-driven compositor layout via generic node view data channel Implement server-driven compositor layout so the backend is the authoritative source of truth for compositor layout in Monitor view (live pipeline), while keeping client-side logic for Design view. Phase 1 — Generic Node View Data Channel: - Add NodeViewDataUpdate type and emit_view_data() helper in core - Wire view_data_tx into NodeContext for optional per-node emission - Add SubscribeViewData/GetNodeViewData to engine query messages - Store + broadcast view data in DynamicEngine (same pattern as stats) - Expose subscribe_view_data()/get_node_view_data() on engine handle - Add NodeViewDataUpdated variant to API EventPayload - Forward view data events in session.rs Phase 2 — Compositor Layout Emission: - Define ResolvedLayer, ResolvedOverlay, CompositorLayout types using SmallVec<[T; 8]> for stack allocation in common case - Add build_layout() helper on CompositorNode - Emit layout via view data channel on config change (dirty flag) - Use PartialEq to skip emission when layout hasn't changed Phase 3 — Frontend Consumption: - Add nodeViewData to SessionData in sessionStore.ts - Handle nodeviewdataupdated event in websocket.ts - Subscribe to server layout in useCompositorLayers.ts (Monitor view) with drag reconciliation (ignore server during drag, accept after) - Add optional serverHeight prop to TextOverlayLayer to bypass client-side hidden-span measurement when server provides height Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: fix formatting (cargo fmt + prettier) Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: add live-node guard to handle_view_data_update + regenerate api-types.ts - Add live_nodes check before processing view data updates to prevent stale data re-insertion after node removal (matches handle_state_update and handle_stats_update patterns) - Regenerate TypeScript types so NodeViewDataUpdated event is included in the WsEventPayload union type Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: aspect ratio stretching in layout emission + session reappearance after delete - build_layout(): apply fit_rect_preserving_aspect() when aspect_fit=true so emitted layout matches actual rendered dimensions (fixes PiP stretching) - sessionStore: updateNodeState/Stats/ViewData and setConnected no longer auto-create session entries for unknown sessions, preventing destroyed sessions from being re-created by late-arriving events - Add initSession() action for explicit session initialization - Update tests to match new behavior Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * fix: opacity clobbering in server layout effect + eliminate perf regression - Video layers: preserve original opacity when visible=false to prevent visibility toggle from getting stuck at opacity=0 - Text overlays: apply same opacity preservation logic - Image overlays: apply same opacity preservation logic - Refactor server layout subscription to use external Zustand subscribe() instead of store selector hook, eliminating unnecessary React re-renders (556 → 383 renders, within 500 budget) Co-Authored-By: Claudio Costa <cstcld91@gmail.com> * style: format useCompositorLayers.ts Co-Authored-By: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: StreamKit Devin <devin@streamkit.dev> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat: more perf work on video compositor * address review
Add mirror_horizontal and mirror_vertical boolean fields to all layer types (video, text overlay, image overlay). Mirroring is applied after rotation in the source coordinate mapping of blit functions. - Add mirror fields to OverlayTransform, LayerConfig, DecodedOverlay, LayerSnapshot, BlitItem, and ResolvedSlotConfig - Apply mirror transform in scale_blit_rgba and scale_blit_rgba_rotated across all code paths (AVX2, SSE2, scalar) - Update benchmark pipeline for new mirror parameters Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Text layers: add resize handles and auto-adjust fontSize on resize to maintain aspect ratio (no stretching) - Session deletion: optimistically remove destroyed sessions from query cache before re-fetching to prevent brief reappearance - Mirror UI: add MirrorControl toggle buttons (horizontal/vertical) to compositor inspector panel; apply scaleX(-1)/scaleY(-1) CSS transforms on canvas preview layers - Frontend state: add mirrorHorizontal/mirrorVertical to LayerState, TextOverlayState, ImageOverlayState; update parse/serialize/merge - E2E: extend monitor tests to cover session deletion persistence Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
- Move shutdown_and_wait() to background tokio::spawn in both WS and HTTP delete handlers so the SessionDestroyed event and response are sent immediately (the 10s shutdown timeout was blocking the WS handler and causing the client's 5s send timeout to fire first) - Fix text overlay YAML format: use flat serde(flatten) format instead of nested 'transform:' key that was silently ignored by deserialization - Make frontend parseTextOverlays/parseImageOverlays handle both flat and legacy nested 'transform:' formats for robustness Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev> Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![staging-devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/870844?s=60&v=4){kind=small}
Comment on lines
+560
to
+591
| for (pin_name, rx) in context.inputs.drain() { | ||
| let is_video = context.input_types.get(&pin_name).is_some_and(|ty| { | ||
| matches!(ty, PacketType::EncodedVideo(_) | PacketType::RawVideo(_)) | ||
| }); | ||
|
|
||
| if is_video { | ||
| if video_rx.is_some() { | ||
| let err_msg = format!( | ||
| "WebMMuxerNode: multiple video inputs detected (pin '{pin_name}'). \ | ||
| Only one video track is supported." | ||
| ); | ||
| state_helpers::emit_failed(&context.state_tx, &node_name, &err_msg); | ||
| return Err(StreamKitError::Runtime(err_msg)); | ||
| } | ||
| tracing::info!( | ||
| "WebMMuxerNode: pin '{pin_name}' classified as VIDEO (from connection type)" | ||
| ); | ||
| video_rx = Some(rx); | ||
| } else { | ||
| if audio_rx.is_some() { | ||
| let err_msg = format!( | ||
| "WebMMuxerNode: multiple audio inputs detected (pin '{pin_name}'). \ | ||
| Only one audio track is supported." | ||
| ); | ||
| state_helpers::emit_failed(&context.state_tx, &node_name, &err_msg); | ||
| return Err(StreamKitError::Runtime(err_msg)); | ||
| } | ||
| tracing::info!( | ||
| "WebMMuxerNode: pin '{pin_name}' classified as AUDIO (from connection type)" | ||
| ); | ||
| audio_rx = Some(rx); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
🚩 WebM muxer input classification defaults to audio when input_types is empty
In crates/nodes/src/containers/webm.rs:560-591, the input classification loop checks context.input_types.get(&pin_name) to determine if a pin carries video. In dynamic pipelines, input_types is always empty (crates/engine/src/dynamic_actor.rs:548: input_types: HashMap::new()), so is_video is always false and ALL inputs are classified as audio. This means the WebM muxer cannot correctly classify video inputs in dynamic mode. Currently this is not triggered because the WebM muxer is only used in oneshot pipelines (where the graph builder at crates/engine/src/graph_builder.rs:303 populates input_types). Dynamic pipelines use MoQ transport for video output. However, if someone wires a WebM muxer in a dynamic pipeline, video inputs would be silently treated as audio, likely producing corrupted output.
Was this helpful? React with 👍 or 👎 to provide feedback.
Debug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(video): compositor, text resize, mirroring, and pipeline deletion fix
Summary
This PR merges the
videofeature branch, adding a full video compositing system with several enhancements and bug fixes built on top:Video Compositor (core feature)
video::compositornode with multi-layer compositing (video inputs, text overlays, image overlays)Text Layer Resizing
fontSizeproportionally to maintain aspect ratio (no stretching)Layer Mirroring
mirror_horizontal/mirror_verticalbooleans on all layer types (video, text, image)scaleX(-1)/scaleY(-1)on canvas preview, toggle buttons in inspectorPipeline Deletion Race Condition Fix
shutdown_and_wait()(10s timeout) blocked the WS handler before broadcastingSessionDestroyed— the client's 5s send timeout expired first, causing the session to briefly reappearshutdown_and_wait()in a backgroundtokio::spawn. Applied to both WS and HTTP delete handlersText Overlay Not Rendering on Pipeline Start
transform:key, butTextOverlayConfiguses#[serde(flatten)]which expects fields at top level — nested key was silently ignored, giving default (0,0) rectReview & Testing Checklist for Human
tokio::spawnforshutdown_and_wait()inwebsocket_handlers.rs:242andserver.rs:1887doesn't leak resources if the server shuts down before the background task completes. Check that theSessionis fully moved into the spawned future.pixel_ops/blit.rs— the mirror transform is applied after rotation inverse-mapping but before pixel read. Verifyisx = sw - 1 - isx/isy = sh - 1 - isylogic is correct for all rotation angles and edge cases (0-width, boundary pixels).video_moq_webcam_pip.ymlround-trips correctly through backend serialize → deserialize and that the frontend fallback for nestedtransform:inuseCompositorLayers.ts:208doesn't mask future config bugs.SK_SERVER__MOQ_GATEWAY_URL=http://127.0.0.1:4545/moq SK_SERVER__ADDRESS=127.0.0.1:4545 just skit), create a Webcam PiP session from Stream view, verify text overlay renders immediately (bottom-left corner), navigate to Monitor, delete the session, confirm it stays gone for 10+ seconds. Also test mirror toggles and text layer resize handles in the compositor canvas.Notes
libvpx-dev(added to workflow)