Performance-driven Takes (#65) #122

Merged

crowmaki merged 1 commit from issue-65-performance-driven-takes into main

2026-10-09 10:47:51 +00:00

crowmaki commented

2026-10-09 10:46:13 +00:00

Owner

Closes #65. Part of #24.

What it does

A Shot whose Lines the Scene's Performance all covers is now driven by it:

Shot length follows the Performance marks of its Lines, plus the usual handles and frame grid.
The slice. A Take gets the Performance audio from the first Line's mark minus the head handle, for exactly the Take's frame count. Audio before the Performance starts or after it ends is silence. The slice is sent in two ways:
- as forced audio, the job builder's third structural edit. VAEEncodeAudio encodes the slice, a 0 noise mask keeps it as it is, and it is joined to the video half of H3's latent at the sampler. The splice point is declared in the h3.draft and h3.final manifests (forced_audio).
- as a fully_copy audio slot. The audio Slot list holds only "Performance vN", always sent and not editable. Voice references drop out.
Prompt. retention_analysis gains <Audio N>: fully_copy - the dialogue of this shot, ….
Provenance. The Take records the Performance version and the slice's start and end (a new performance kind of take_input). It also snapshots the Performance marks inside the slice. Those marks become its default Line marks, so later mark edits never move them. Nudging and resetting work as before; a reset goes back to the snapshot.
Straddle block. A Shot over both covered and uncovered Lines can't be queued: "split the shot or extend the performance". The Shot row offers a split where coverage changes. A length override doesn't lift the block.
Uncovered Shots. A Shot entirely over uncovered Lines renders as if there were no Performance.
Per-Scene switch. "Drives Takes" in the Animatic's Performance bar is an undoable edit. When off, the Scene's Takes generate their own dialogue again. Versions and marks are untouched.
Staleness. A new Performance reason group:
- "Performance vN: new version, was vM"
- "Performance vN: marks of Lines a–b moved", which fires only when this Shot's slice moved
- "Performance no longer sent", when the switch is off
The input table shows the slot, the version and the slice span.

Schema V36 adds scene.performance_off, rebuilds take_input with performance_id / slice_from_ms / slice_to_ms, and adds take_performance_mark.

Decisions worth a look

The switch only affects Takes (slots, Shot length, staleness). The Animatic keeps timing the Scene by the Performance and keeps playing it under Boards. While it is off, the Shot lengths go back to estimates, so they can differ from the Animatic's timing.
Handles hold real audio. The slice's handles contain the neighbouring Performance audio, not silence.
Promotion adds no forced audio. The draft's audio latent already holds the forced audio, and the promotion graph keeps it with its 0 mask.
Forced-audio nodes. The forced-audio edit adds LTXVSeparateAVLatent → LoadAudio → VAEEncodeAudio → SolidMask/SetLatentNoiseMask → LTXVConcatAVLatent. This is the same masking pattern h3.promotion uses. Not yet run against a real ComfyUI/H3; please confirm H3 accepts the joined latent at 13.latent_image.
No workflow version bump. The templates are unchanged and non-driven jobs are identical, so the manifests' version was not bumped.

Tests

crates/studio-core/tests/performance_takes.rs (new) covers:
- Shot timing
- the slice's bytes and padding
- the graph wiring (draft and single-pass final)
- the fully_copy prompt entry and slot list
- uncovered Shots
- the straddle block
- the switch with undo
- snapshotted marks
- staleness scope (a mark edit makes only the moved Shot's Take stale; new version; switch)
performance_slice.rs has unit tests for drive, slice and padding.
cargo test --workspace --features studio-core/test-support, bun test src and bun run check pass.

Screenshots

A driven Shot: "Performance v1" badge, and the audio list holds only the forced slice.

A Shot over covered and uncovered Lines is blocked, with a split offer.

The Animatic's Performance bar with the new "Drives Takes" switch, on and off.

Closes #65. Part of #24. ## What it does A Shot whose Lines the Scene's Performance all covers is now **driven** by it: - **Shot length** follows the Performance marks of its Lines, plus the usual handles and frame grid. - **The slice.** A Take gets the Performance audio from the first Line's mark minus the head handle, for exactly the Take's frame count. Audio before the Performance starts or after it ends is silence. The slice is sent in two ways: - as **forced audio**, the job builder's third structural edit. `VAEEncodeAudio` encodes the slice, a 0 noise mask keeps it as it is, and it is joined to the video half of H3's latent at the sampler. The splice point is declared in the `h3.draft` and `h3.final` manifests (`forced_audio`). - as a **`fully_copy` audio slot**. The audio Slot list holds only "Performance vN", always sent and not editable. Voice references drop out. - **Prompt.** `retention_analysis` gains `<Audio N>: fully_copy - the dialogue of this shot, …`. - **Provenance.** The Take records the Performance version and the slice's start and end (a new `performance` kind of `take_input`). It also snapshots the Performance marks inside the slice. Those marks become its default Line marks, so later mark edits never move them. Nudging and resetting work as before; a reset goes back to the snapshot. - **Straddle block.** A Shot over both covered and uncovered Lines can't be queued: "split the shot or extend the performance". The Shot row offers a split where coverage changes. A length override doesn't lift the block. - **Uncovered Shots.** A Shot entirely over uncovered Lines renders as if there were no Performance. - **Per-Scene switch.** "Drives Takes" in the Animatic's Performance bar is an undoable edit. When off, the Scene's Takes generate their own dialogue again. Versions and marks are untouched. - **Staleness.** A new Performance reason group: - "Performance vN: new version, was vM" - "Performance vN: marks of Lines a–b moved", which fires only when *this* Shot's slice moved - "Performance no longer sent", when the switch is off The input table shows the slot, the version and the slice span. Schema V36 adds `scene.performance_off`, rebuilds `take_input` with `performance_id` / `slice_from_ms` / `slice_to_ms`, and adds `take_performance_mark`. ## Decisions worth a look - **The switch only affects Takes** (slots, Shot length, staleness). The Animatic keeps timing the Scene by the Performance and keeps playing it under Boards. While it is off, the Shot lengths go back to estimates, so they can differ from the Animatic's timing. - **Handles hold real audio.** The slice's handles contain the neighbouring Performance audio, not silence. - **Promotion adds no forced audio.** The draft's audio latent already holds the forced audio, and the promotion graph keeps it with its 0 mask. - **Forced-audio nodes.** The forced-audio edit adds `LTXVSeparateAVLatent` → `LoadAudio` → `VAEEncodeAudio` → `SolidMask`/`SetLatentNoiseMask` → `LTXVConcatAVLatent`. This is the same masking pattern `h3.promotion` uses. **Not yet run against a real ComfyUI/H3**; please confirm H3 accepts the joined latent at `13.latent_image`. - **No workflow version bump.** The templates are unchanged and non-driven jobs are identical, so the manifests' `version` was not bumped. ## Tests - `crates/studio-core/tests/performance_takes.rs` (new) covers: - Shot timing - the slice's bytes and padding - the graph wiring (draft and single-pass final) - the fully_copy prompt entry and slot list - uncovered Shots - the straddle block - the switch with undo - snapshotted marks - staleness scope (a mark edit makes only the moved Shot's Take stale; new version; switch) - `performance_slice.rs` has unit tests for drive, slice and padding. - `cargo test --workspace --features studio-core/test-support`, `bun test src` and `bun run check` pass. ## Screenshots A driven Shot: "Performance v1" badge, and the audio list holds only the forced slice. ![Driven shot](https://git.crowmaki.me/attachments/065a0a7d-a479-4382-864c-8a9d4b235ef7) A Shot over covered and uncovered Lines is blocked, with a split offer. ![Straddling shot](https://git.crowmaki.me/attachments/d5ef9497-b987-4a70-94c8-a7de85889631) The Animatic's Performance bar with the new "Drives Takes" switch, on and off. ![Switch on](https://git.crowmaki.me/attachments/e8f181ba-1939-4a22-99ee-ae989e169d1a) ![Switch off](https://git.crowmaki.me/attachments/984c0b1c-aa09-464d-9c80-cc0edf2af4bc)

1-shot-driven.png

66 KiB

2-shot-straddles.png

69 KiB

3-animatic-on.png

45 KiB

4-animatic-off.png

45 KiB

crowmaki added 1 commit

2026-10-09 10:46:13 +00:00

Performance-driven Takes (#65 ) 01ec2a6eb9

A Shot whose Lines the Scene's Performance all covers is timed by its
marks and sends a Take its slice of the Performance: as forced audio
(VAEEncodeAudio with a 0 noise mask, joined to the video half of H3's
latent at the sampler) and as a fully_copy audio slot. Voice references
drop. The Take records the version and slice, and starts with the
Performance marks inside it as its Line marks.

A Shot straddling covered and uncovered Lines blocks queuing with a split
offer. A per-Scene switch turns the Performance off for Takes, undoably,
keeping its marks. Staleness gains Performance reasons: a new version,
marks of the Shot's Lines moved (only when its slice moved), and the
slice no longer sent.

Co-Authored-By: Claude <noreply@anthropic.com>

crowmaki merged commit 2e194ce443 into main

2026-10-09 10:47:51 +00:00

crowmaki deleted branch issue-65-performance-driven-takes

2026-10-09 10:47:51 +00:00

crowmaki referenced this pull request from a commit

2026-10-09 10:47:52 +00:00

Merge pull request 'Performance-driven Takes (#65)' (#122) from issue-65-performance-driven-takes into main