Performance import and marking (#63) #120

Merged

crowmaki merged 2 commits from issue-63-performance-import into main

2026-10-09 09:55:29 +00:00

crowmaki commented

2026-10-09 09:53:47 +00:00

Owner

Closes #63. Part of #24.

What

A filmmaker can lock a Scene's dialogue timing with recorded audio.

Import audio for a contiguous run of a Scene's Lines (all of them by default). Each import is a new, immutable Performance version. The audio is stored as 48 kHz FLAC, with a 10 ms peak waveform and the islands of speech.
Auto-placement: ffmpeg silencedetect (from the sidecar) finds the speech. The islands go in order to the dialogue units, merged or split in proportion to their estimates. Marks are a contiguous partition:
- a pause goes to the earlier Line;
- action Lines between dialogue split the silence;
- leading and trailing action Lines take the head and tail;
- a dual dialogue pair is one span.
Marks are undoable edits on the newest version: drag a Line mark, or nudge it with the arrow keys. Re-run auto-placement replaces every mark as one undoable edit. Old versions keep their marks as history and can't be edited.
Coverage: performance_mark.covered per Line per version. A Line outside the imported run has no row and stays uncovered.
The Cut: covered Lines take their durations from the Performance (LineTiming::Performance), and every event except a Take event plays the Performance's audio under it.
Performance lane on the Animatic:
- the waveform, with a draggable Line mark per unit;
- a warning on Lines under 1 s;
- shading under uncovered Lines;
- a ghost lane with the previous version's marks;
- a header with the version history, a ghost switch, Re-run auto-placement and Import.

Schema step V34 adds performance, performance_source and performance_mark.

Left for later tickets:

Bounce (#64).
Everything that sends a slice to a Take (#65): snapshotted Take marks, the straddle block, staleness by slice, performance_off.

Notes for review

Where a Line has a Performance, the Performance times it ahead of a circled Take's Line marks. A Take event plays from its first Line mark, for the Performance's duration. Until #65 makes Takes Performance-driven, the two can disagree.
Auto-placement either merges extra islands or splits too few. It never does both in one Scene, and an action Line inside a split island gets no room (it is flagged under 1 s). Correcting these by hand is the intended path, per #23.
The weights are the Lines' durations now (a manual override, else the text estimate).

Tests

performance_marks unit tests cover silence parsing and the partition rules.
tests/performance.rs imports ffmpeg-generated fixture audio and covers the marks, the run, the Cut timing and audio clips, undo, re-run, versions and the ghost, and the short-Line flag.
performance.test.ts covers the lane helpers.
cargo test --workspace --all-features, bun test src and bun run check all pass.

Screenshots

Before any import:

Imported, with marks auto-placed on the waveform:

After dragging two Line marks. The short action Line gets a warning, and the selected Line says it is timed by the performance:

A second version, with v1's marks in the ghost lane:

Version history:

Closes #63. Part of #24. ## What A filmmaker can lock a Scene's dialogue timing with recorded audio. - **Import** audio for a contiguous run of a Scene's Lines (all of them by default). Each import is a new, immutable Performance version. The audio is stored as 48 kHz FLAC, with a 10 ms peak waveform and the islands of speech. - **Auto-placement**: ffmpeg `silencedetect` (from the sidecar) finds the speech. The islands go in order to the dialogue units, merged or split in proportion to their estimates. Marks are a contiguous partition: - a pause goes to the earlier Line; - action Lines between dialogue split the silence; - leading and trailing action Lines take the head and tail; - a dual dialogue pair is one span. - **Marks** are undoable edits on the newest version: drag a Line mark, or nudge it with the arrow keys. Re-run auto-placement replaces every mark as one undoable edit. Old versions keep their marks as history and can't be edited. - **Coverage**: `performance_mark.covered` per Line per version. A Line outside the imported run has no row and stays uncovered. - **The Cut**: covered Lines take their durations from the Performance (`LineTiming::Performance`), and every event except a Take event plays the Performance's audio under it. - **Performance lane** on the Animatic: - the waveform, with a draggable Line mark per unit; - a warning on Lines under 1 s; - shading under uncovered Lines; - a ghost lane with the previous version's marks; - a header with the version history, a ghost switch, Re-run auto-placement and Import. Schema step V34 adds `performance`, `performance_source` and `performance_mark`. Left for later tickets: - Bounce (#64). - Everything that sends a slice to a Take (#65): snapshotted Take marks, the straddle block, staleness by slice, `performance_off`. ## Notes for review - Where a Line has a Performance, the Performance times it ahead of a circled Take's Line marks. A Take event plays from its first Line mark, for the Performance's duration. Until #65 makes Takes Performance-driven, the two can disagree. - Auto-placement either merges extra islands or splits too few. It never does both in one Scene, and an action Line inside a split island gets no room (it is flagged under 1 s). Correcting these by hand is the intended path, per #23. - The weights are the Lines' durations now (a manual override, else the text estimate). ## Tests - `performance_marks` unit tests cover silence parsing and the partition rules. - `tests/performance.rs` imports ffmpeg-generated fixture audio and covers the marks, the run, the Cut timing and audio clips, undo, re-run, versions and the ghost, and the short-Line flag. - `performance.test.ts` covers the lane helpers. - `cargo test --workspace --all-features`, `bun test src` and `bun run check` all pass. ## Screenshots Before any import: ![No performance](https://git.crowmaki.me/attachments/b31f3ee9-31c6-4631-93e5-1714f7285916) Imported, with marks auto-placed on the waveform: ![Imported](https://git.crowmaki.me/attachments/a25b328b-c7d5-4687-8317-93225505f927) After dragging two Line marks. The short action Line gets a warning, and the selected Line says it is timed by the performance: ![Marked by hand](https://git.crowmaki.me/attachments/73f2bc9a-5459-4ab7-8845-3877dcf733a8) A second version, with v1's marks in the ghost lane: ![Ghost lane](https://git.crowmaki.me/attachments/3de2c062-f5b2-48cb-8961-7ea4fc50f819) Version history: ![Version history](https://git.crowmaki.me/attachments/ee394cc7-6a0d-43bf-80ee-49b58b94e39a)

0-no-performance.png

49 KiB

1-imported.png

47 KiB

2-marked-by-hand.png

48 KiB

3-ghost-lane.png

48 KiB

4-version-history.png

54 KiB

crowmaki added 2 commits

2026-10-09 09:53:47 +00:00

Performance import and marking (#63 ) 6eec77f064

Import Performance audio for a contiguous run of a Scene's Lines as a new,
immutable Performance version. ffmpeg's silencedetect finds the speech, and
auto-placement partitions the audio into Line marks. Marks are undoable
edits on the newest version, and so is re-running auto-placement.

The Cut times covered Lines from the Performance, and Board events play
its audio. The Animatic gets a Performance lane with the waveform,
draggable boundaries, warnings for Lines under 1 s, shading under
uncovered Lines and a ghost lane of the previous version.

Co-Authored-By: Claude <noreply@anthropic.com>

Address review of Performance import and marking (#63 ) c129763289

Re-run auto-placement reads its Lines the way import does, the lane places
each Line mark at its own unit in the Cut, and the undo doc lists
Performance marks.

Co-Authored-By: Claude <noreply@anthropic.com>

crowmaki merged commit 0245161457 into main

2026-10-09 09:55:29 +00:00

crowmaki deleted branch issue-63-performance-import

2026-10-09 09:55:29 +00:00

crowmaki referenced this pull request from a commit

2026-10-09 09:55:30 +00:00

Merge pull request 'Performance import and marking (#63)' (#120) from issue-63-performance-import into main