Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
e1d5337
chore: update roadmap
streamer45 Jan 24, 2026
fa7865a
feat(video): update packet types, docs, and compatibility rules
streamer45 Jan 24, 2026
b9455d0
feat(video): make raw video layout explicit + enforce aligned buffers
streamer45 Jan 24, 2026
a459202
Merge origin/main into video branch (reconcile recent PRs)
streamkit-devin Feb 14, 2026
4b45cf6
feat(webm): extend muxer with VP9 video track support (PR4)
streamkit-devin Feb 21, 2026
3eb3424
feat: end-to-end video pipeline support
streamkit-devin Feb 22, 2026
5214bfc
fix(ui): video-aware ConvertView for no-input pipelines
streamkit-devin Feb 22, 2026
5802c6d
fix(server): allow generator-only oneshot pipelines without http_input
streamkit-devin Feb 22, 2026
efb0ebb
fix(engine): allow generator-only oneshot pipelines without file_reader
streamkit-devin Feb 22, 2026
85e6a86
fix(nodes): enable video feature (vp9 + colorbars) in default features
streamkit-devin Feb 22, 2026
8b47499
fix: generator pipeline start signals, video-only content-type, and m…
streamkit-devin Feb 22, 2026
58ed8a0
fix
streamer45 Feb 22, 2026
eb0d690
feat: add sweep bar animation to colorbars, skip publish for receive-…
streamkit-devin Feb 22, 2026
cac4c20
perf(vp9): configurable encoder deadline (default realtime), avoid un…
streamkit-devin Feb 22, 2026
a7a3188
style: cargo fmt
streamkit-devin Feb 22, 2026
d054409
test(e2e): add video pipeline tests for convert and MoQ stream views
streamkit-devin Feb 22, 2026
0d63a3c
fix: correct webm_muxer pin name in mixing pipeline and convert butto…
streamkit-devin Feb 22, 2026
9774d3e
refactor(webm-muxer): generic input pins with runtime media type dete…
streamkit-devin Feb 23, 2026
1ea47ff
style: cargo fmt
streamkit-devin Feb 23, 2026
27fad06
refactor(webm-muxer): connection-time type detection via NodeContext.…
streamkit-devin Feb 23, 2026
43e3a9d
feat(compositor): add video compositor node with dynamic inputs, over…
streamkit-devin Feb 23, 2026
d3f44b5
style: cargo fmt
streamkit-devin Feb 23, 2026
e22e9e6
fix(compositor): address review findings and add sample pipeline
streamkit-devin Feb 23, 2026
7f7d267
fix(compositor): use single needs variant in sample pipeline YAML
streamkit-devin Feb 23, 2026
f672387
fix(compositor): remove deeply nested params from sample YAML
streamkit-devin Feb 23, 2026
1a10a66
fix(compositor): add num_inputs for static pin pre-creation in onesho…
streamkit-devin Feb 23, 2026
64a99ab
feat(compositor): accept I420 inputs and configurable output format
streamkit-devin Feb 23, 2026
c04f826
fix(compositor): process every frame instead of draining to latest
streamkit-devin Feb 23, 2026
b8cc54a
feat(compositor): auto-PiP positioning and two-input sample pipeline
streamkit-devin Feb 23, 2026
ec0953d
perf(compositor): move all pixel format conversions into spawn_blocking
streamkit-devin Feb 23, 2026
aa9cc75
perf(compositor): parallelize with rayon and use persistent blocking …
streamkit-devin Feb 24, 2026
58dd39a
refactor(compositor): modularize into config, overlay, pixel_ops, and…
streamkit-devin Feb 24, 2026
91d0c92
perf(compositor): 5 high-impact video compositing optimizations
streamkit-devin Feb 24, 2026
27fe2f2
style: apply cargo fmt formatting
streamkit-devin Feb 24, 2026
a7d2bd5
perf(compositor): fix regression — replace broken chunking with slice…
streamkit-devin Feb 24, 2026
f41213d
style: apply cargo fmt formatting
streamkit-devin Feb 24, 2026
972fbd9
perf(compositor): revert chunks_exact to simple for-loops
streamkit-devin Feb 24, 2026
5ed214e
perf(compositor): eliminate double-copy in I420 output path
streamkit-devin Feb 24, 2026
49d6163
bench: add compositor pipeline benchmark for profiling
streamkit-devin Feb 24, 2026
1736790
fix: resolve clippy lint errors in video nodes
streamkit-devin Feb 24, 2026
5dc1eab
fix: resolve remaining clippy lint errors in video nodes
streamkit-devin Feb 24, 2026
e8b0c06
fix: make lint pass after metadata updates
streamkit-devin Feb 24, 2026
d629020
chore: update native plugin lockfiles
streamkit-devin Feb 24, 2026
3638d25
Merge remote-tracking branch 'origin/main' into video
streamer45 Feb 24, 2026
ed79257
fix(webm): skip intermediate flushes in File mode to prevent finalize…
streamkit-devin Feb 24, 2026
b401f14
fix(webm): use Live mode for VP9 mux test to avoid unbounded memory
streamkit-devin Feb 24, 2026
1c1f49c
style: apply cargo fmt formatting
streamkit-devin Feb 24, 2026
28081eb
fix(compositor): handle non-video packets and single channel close in…
streamkit-devin Feb 24, 2026
61964bb
fix(webm): restore streaming-mode guard in flush_output
streamkit-devin Feb 24, 2026
4d78b76
fix(moq): remove hardcoded catalog dimensions and add clean shutdown
streamkit-devin Feb 24, 2026
1a6fa04
fix(vp9): improve encoder/decoder allocations and add shutdown comments
streamkit-devin Feb 24, 2026
590d7d8
refactor(video): extract shared parse_pixel_format utility
streamkit-devin Feb 24, 2026
0b2f26b
fix: sweep bar clipping, WebM auto-detect dims, output filename
streamkit-devin Feb 24, 2026
7c074c8
style: apply cargo fmt to webm muxer
streamkit-devin Feb 24, 2026
2acabb9
perf: collapse SharedPacketBuffer mutexes, bump pool max, zero-alloc …
streamkit-devin Feb 24, 2026
9bcb5c0
style: apply cargo fmt
streamkit-devin Feb 24, 2026
3c3fa09
fix(sample): add pacer node to video compositor demo for real-time pl…
streamkit-devin Feb 24, 2026
1ddf0de
fix(engine): walk connection graph backwards for content-type resolution
streamkit-devin Feb 24, 2026
3322693
fix(compositor): sort input slots by pin name for deterministic layer…
streamkit-devin Feb 24, 2026
3309ea6
feat(compositor): add z_index to LayerConfig for explicit layer stack…
streamkit-devin Feb 24, 2026
cc6ba5b
style: apply cargo fmt to compositor z_index changes
streamkit-devin Feb 24, 2026
ad537a5
perf: review fixes — temp file for WebM File mode, Arc unwrap, rayon …
streamkit-devin Feb 25, 2026
7985ba5
feat(compositor): sweep_bar toggle, fontdue text rendering, rotation,…
streamkit-devin Feb 25, 2026
33a2357
style: apply cargo fmt formatting
streamkit-devin Feb 25, 2026
d07ee58
feat(demo): add text overlay layer with bundled DejaVu Sans font
streamkit-devin Feb 25, 2026
babe41d
fix: work around serde_saphyr untagged enum limitation for nested YAML
streamkit-devin Feb 25, 2026
3594e7a
style: apply cargo fmt to server.rs
streamkit-devin Feb 25, 2026
f93fba4
fix(demo): move PiP overlay positioning to the left
streamkit-devin Feb 25, 2026
a67ac22
refactor(colorbars): remove sweep_bar parameter entirely
streamkit-devin Feb 25, 2026
bded9a6
feat(compositor): add draw_time option with millisecond precision
streamkit-devin Feb 25, 2026
1636311
style: apply cargo fmt to draw_time changes
streamkit-devin Feb 25, 2026
5c3fd5d
fix(compositor): add edge anti-aliasing for rotated layers
streamkit-devin Feb 25, 2026
935c2fb
refactor(colorbars): move draw_time from compositor to colorbars gene…
streamkit-devin Feb 25, 2026
cd8d1d1
refactor: deduplicate and improve video subsystem code quality
streamkit-devin Feb 25, 2026
ca785e4
style: apply cargo fmt
streamkit-devin Feb 25, 2026
f7594a8
feat(ui): add compositor node UI with draggable layer canvas
streamkit-devin Feb 25, 2026
278e788
fix(ui): collapse unscaled height in compositor canvas via negative m…
streamkit-devin Feb 25, 2026
79c63c5
fix(ui): map video::compositor type in YAML pipeline parser
streamkit-devin Feb 25, 2026
3eb42d8
fix(ui): enable compositor layer interactions in Design View
streamkit-devin Feb 27, 2026
e63ae1d
style: format useCompositorLayers.ts
streamkit-devin Feb 27, 2026
49f3ae5
feat: add Video Compositor (MoQ Stream) pipeline template
streamkit-devin Feb 28, 2026
b2cc567
feat(ui): Complete compositor UX improvements
streamkit-devin Feb 28, 2026
49e07a9
fix(ui): prevent compositor node overlap in auto-layout
streamkit-devin Feb 28, 2026
4fca21e
feat(ui): compositor UX improvements - layer rendering, floating prev…
streamkit-devin Feb 28, 2026
28d34a6
perf(compositor,vp9): eliminate format bounce and add SSE2 SIMD (#62)
staging-devin-ai-integration[bot] Feb 28, 2026
ae602bc
feat: NV12 as default video format (#63)
staging-devin-ai-integration[bot] Feb 28, 2026
7e55230
perf: enable thin LTO, codegen-units=1, and target-cpu=native for pro…
staging-devin-ai-integration[bot] Feb 28, 2026
54b87cb
perf(compositor): implement findings 1+4, 2, 5, and 3 for video compo…
staging-devin-ai-integration[bot] Feb 28, 2026
09bc004
style(compositor): fix clippy and rustfmt lint issues in SIMD kernels
streamkit-devin Mar 1, 2026
c104b2c
perf(compositor): cache available_parallelism in LazyLock for rayon_c…
streamkit-devin Mar 1, 2026
80e55b1
style(compositor): apply rustfmt to LazyLock closure
streamkit-devin Mar 1, 2026
379f1f4
fix(compositor): correct AVX2 lane-crossing in chroma kernels
streamkit-devin Mar 1, 2026
d99c7f6
feat(colorbars): default output pixel format to RGBA8
streamkit-devin Mar 1, 2026
dd73c0e
perf(compositor): add AVX2 NV12→RGBA8 kernel and hoist CPU feature de…
streamkit-devin Mar 1, 2026
b79689c
perf(compositor): algorithmic optimizations, SSE2 blend + microbenchm…
staging-devin-ai-integration[bot] Mar 1, 2026
7a79b9d
feat(compositor-ui): UX improvements for video compositor (#69)
staging-devin-ai-integration[bot] Mar 1, 2026
2a8207c
fix(compositor-ui): use committedRef to prevent double-fire on Enter+…
staging-devin-ai-integration[bot] Mar 1, 2026
2e355f7
fix(video): preserve aspect ratio in compositor rotation and stream r…
staging-devin-ai-integration[bot] Mar 1, 2026
40213f6
fix(compositor-ui): address 7 UX issues in compositor node (#72)
staging-devin-ai-integration[bot] Mar 1, 2026
4e9b86f
feat(compositor): consolidate overlay transforms + unified z-sorted b…
streamkit-devin Mar 1, 2026
ed0d37b
style: apply prettier formatting to CompositorCanvas.tsx
streamkit-devin Mar 1, 2026
df080ec
fix(compositor): correct SIMD blit double-counting in rotated interio…
streamkit-devin Mar 1, 2026
d150cc0
style: apply rustfmt to pixel_ops.rs
streamkit-devin Mar 1, 2026
6d910a0
fix(compositor-ui): UX polish — padding, snap-to-grid, letterbox full…
streamkit-devin Mar 1, 2026
9507dc8
fix(compositor-ui): preserve position on click-only select, use overl…
streamkit-devin Mar 1, 2026
a846489
fix(nodes,engine,api): address all clippy lint issues on video branch
streamkit-devin Mar 1, 2026
03b3dff
refactor(pixel_ops): modularize into focused files, deduplicate SIMD,…
staging-devin-ai-integration[bot] Mar 1, 2026
c5c8f1b
perf(compositor): AVX2 SIMD kernels, parallel identity blit, overlay …
staging-devin-ai-integration[bot] Mar 1, 2026
2611de6
fix(compositor): use stretch-to-fill consistently in blit paths (#75)
staging-devin-ai-integration[bot] Mar 2, 2026
787e0cb
perf(compositor): fuse YUV output conversion, AVX2 rotated blit inter…
staging-devin-ai-integration[bot] Mar 2, 2026
169dd9a
fix(compositor): improve image overlay quality, caching, aspect ratio…
staging-devin-ai-integration[bot] Mar 3, 2026
9a27b21
hd at least
streamer45 Mar 3, 2026
f2ba317
refactor: consolidate compositor layer UI controls and hook helpers (…
staging-devin-ai-integration[bot] Mar 3, 2026
b8d9051
feat(compositor): text color, font selection, draggable layers, clipp…
staging-devin-ai-integration[bot] Mar 3, 2026
d055856
feat(ui): consolidate duplicated frontend components and hooks (#81)
staging-devin-ai-integration[bot] Mar 4, 2026
097b1bc
perf(compositor): SIMD alpha scan, font cache, fat LTO (#82)
staging-devin-ai-integration[bot] Mar 4, 2026
0a240c0
feat(video): bundle DejaVu fonts at compile time (#83)
staging-devin-ai-integration[bot] Mar 4, 2026
af1f4de
fix: address video branch review findings (#84)
staging-devin-ai-integration[bot] Mar 4, 2026
aa0347b
feat(video): add webcam input support and PiP demo (#85)
staging-devin-ai-integration[bot] Mar 5, 2026
98d6305
fix(nodes): resolve clippy lints and REUSE compliance on video branch…
staging-devin-ai-integration[bot] Mar 6, 2026
915a461
feat: upgrade moq ecosystem dependencies to latest versions (#87)
staging-devin-ai-integration[bot] Mar 6, 2026
dbe7455
feat(moq_peer): type-agnostic pins with runtime media kind inference …
staging-devin-ai-integration[bot] Mar 6, 2026
eb526bd
fix(ui): minimize compositor re-renders during opacity/rotation slide…
staging-devin-ai-integration[bot] Mar 6, 2026
d48fcf1
fix(video): text wrapping, canvas cleanup, and YAML debounce (#90)
staging-devin-ai-integration[bot] Mar 7, 2026
1fd0142
feat(ui): improve compositor node UX (#91)
staging-devin-ai-integration[bot] Mar 7, 2026
5e1d791
feat(video): add pixel_convert node and remove RGBA conversion from V…
staging-devin-ai-integration[bot] Mar 7, 2026
3f51ef3
feat(ui): add two-layer render-performance profiling infrastructure (…
staging-devin-ai-integration[bot] Mar 7, 2026
488326b
feat: server-driven compositor layout via generic node view data chan…
staging-devin-ai-integration[bot] Mar 7, 2026
3cdeba2
feat: more perf work on video compositor (#96)
streamer45 Mar 7, 2026
18c31ec
feat(video): add horizontal/vertical mirror support to compositor layers
streamkit-devin Mar 7, 2026
4508676
style: apply cargo fmt to compositor mirror changes
streamkit-devin Mar 7, 2026
ed53eac
feat(ui): text layer resize, session deletion race fix, mirror UI
streamkit-devin Mar 7, 2026
aad560f
fix: add missing mirror fields to OverlayTransform in test
streamkit-devin Mar 7, 2026
cb87d1a
fix: pipeline deletion race condition and text overlay rendering
streamkit-devin Mar 7, 2026
7a38fc8
style: format useCompositorLayers.ts
streamkit-devin Mar 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/skit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ jobs:
bun install --frozen-lockfile
bun run build

- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libvpx-dev

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@master
with:
Expand Down Expand Up @@ -89,6 +92,9 @@ jobs:
bun install --frozen-lockfile
bun run build

- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libvpx-dev

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@master
with:
Expand Down Expand Up @@ -129,6 +135,9 @@ jobs:
bun install --frozen-lockfile
bun run build

- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libvpx-dev

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@master
with:
Expand Down
105 changes: 105 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,111 @@ Agent-assisted contributions are welcome, but should be **supervised** and **rev
- Follow `CONTRIBUTING.md` (DCO sign-off, Conventional Commits, SPDX headers where applicable).
- **Linting discipline**: Do not blindly suppress lint warnings or errors with ignore/exception rules. Instead, consider refactoring or improving the code to address the underlying issue. If an exception is truly necessary, it **must** include a comment explaining the rationale.

## Running E2E tests

End-to-end tests live in `e2e/` and use Playwright (Chromium, headless).

1. **Build the UI** and **start the server** in one terminal:

```bash
just build-ui && SK_SERVER__MOQ_GATEWAY_URL=http://127.0.0.1:4545/moq SK_SERVER__ADDRESS=127.0.0.1:4545 just skit
```

2. **Run the tests** in a second terminal:

```bash
just e2e-external http://localhost:4545
```

### Headless-browser pitfalls

- Playwright runs headless Chromium with a default 1280×720 viewport.
Elements rendered below the fold are **not visible** to
`IntersectionObserver`. If a test relies on an element being observed
(e.g. the `<canvas>` used by the MoQ video renderer), scroll it into
view first:

```ts
const canvas = page.locator('canvas');
await canvas.scrollIntoViewIfNeeded();
```

- The `@moq/watch` `Video.Renderer` enables the `Video.Decoder` (and
therefore the `video/data` MoQ subscription) **only** when the canvas is
intersecting. Forgetting to scroll will result in a permanently black
canvas.

## Render performance profiling

StreamKit ships a two-layer profiling infrastructure for detecting render
regressions — particularly **cascade re-renders** where a slider interaction
(opacity, rotation) triggers expensive re-renders in unrelated memoized
components (`UnifiedLayerList`, `OpacityControl`, `RotationControl`, etc.).

### When to use this

- **After touching compositor hooks or components** (`useCompositorLayers`,
`CompositorNode`, or any `React.memo`'d sub-component): run the perf tests
to verify you haven't broken memoization barriers.
- **When optimising render performance**: use the baseline comparison to
measure before/after render counts and durations.
- **In CI**: Layer 1 tests run automatically via `just perf-ui` and will fail
if render counts regress beyond the 2σ threshold stored in the baseline.

### Layer 1 — Component-level regression tests (Vitest)

Fast, deterministic tests that measure hook/component render counts in
happy-dom. No browser required.

```bash
just perf-ui # runs all *.perf.test.* files
```

Key files:

| File | Purpose |
|------|---------|
| `ui/src/test/perf/measure.ts` | `measureRenders()` (components) and `measureHookRenders()` (hooks) |
| `ui/src/test/perf/compare.ts` | Baseline read/write, 2σ comparison, report formatting |
| `ui/src/hooks/useCompositorLayers.render-perf.test.ts` | Cascade re-render regression tests |
| `perf-baselines.json` (repo root) | Baseline snapshot — committed to track regressions over time |

**Cascade detection pattern**: the render-perf tests simulate rapid slider
drags (20 ticks of opacity/rotation) and assert that total render count stays
within a budget (currently ≤ 30). If callback references become unstable
(e.g. `layers` array in deps instead of `selectedLayerKind`), React.memo
barriers break and the render count will blow past the budget, failing the
test.

### Layer 2 — Interaction-level profiling (Playwright + React.Profiler)

Real-browser profiling for dev builds. Components wrapped with
`React.Profiler` push metrics to `window.__PERF_DATA__` which Playwright
tests can read via `page.evaluate()`.

```bash
just perf-e2e # requires: just skit + just ui (dev server at :3045)
```

Key files:

| File | Purpose |
|------|---------|
| `ui/src/perf/profiler.ts` | Dev-only `PerfProfiler` wrapper + `window.__PERF_DATA__` store |
| `e2e/tests/perf-helpers.ts` | `capturePerfData()` / `resetPerfData()` Playwright utilities |
| `e2e/tests/compositor-perf.spec.ts` | E2E test: creates PiP session, drags all sliders, asserts render budget |

Use Layer 2 when you need real paint/layout timing or want to profile
interactions end-to-end with actual browser rendering.

### Updating the baseline

Run `just perf-ui` — the last test in the render-perf suite writes a fresh
`perf-baselines.json` (gated behind `UPDATE_PERF_BASELINE=1`, which the
`test:perf` script sets automatically). Regular `just test-ui` runs compare
against the baseline but never overwrite it. Commit the updated baseline
alongside your changes so future runs compare against the new numbers.

## Docker notes

- Official images are built from `Dockerfile` (CPU) and `Dockerfile.gpu` (GPU-tagged) via `.github/workflows/docker.yml`.
Expand Down
Loading
Loading