Marking Lines in a Performance #23

New issue

Closed

opened 2026-10-06 07:32:31 +00:00 by crowmaki · 1 comment

crowmaki commented

2026-10-06 07:32:31 +00:00

Owner

Part of #1

Question

How are a Scene's Lines marked in its Performance, so each Line gets its time span? Is it marked by hand on a waveform, derived (e.g. by forced alignment or silence detection), or derived then corrected? How are action Lines and gaps between dialogue placed? How do dual-dialogue simultaneous groups overlap? What happens to the marks when the Performance is replaced? From "Animatic duration rules" (#8): Performance marks are the top source of Line durations and the default Line marks for Takes conditioned on it.

Part of #1 ## Question How are a Scene's Lines marked in its Performance, so each Line gets its time span? Is it marked by hand on a waveform, derived (e.g. by forced alignment or silence detection), or derived then corrected? How are action Lines and gaps between dialogue placed? How do dual-dialogue simultaneous groups overlap? What happens to the marks when the Performance is replaced? From "Animatic duration rules" (#8): Performance marks are the top source of Line durations and the default Line marks for Takes conditioned on it.

crowmaki added the

wayfinder:grilling

label

2026-10-06 07:32:36 +00:00

crowmaki referenced this issue

2026-10-06 07:32:50 +00:00

Wayfinder: AI Studio end-to-end slice spec #1

crowmaki self-assigned this

2026-10-07 07:01:08 +00:00

crowmaki commented

2026-10-07 07:14:03 +00:00

Author

Owner

Resolution

Sources. A Performance is either a Bounce of the Scene's Cut audio or imported audio. A source either supplies its own Line marks (a Bounce; a future per-Line TTS render) or needs marking (an import). Alternate voice sources such as local TTS are out of scope, but performance_source must stay open to new source kinds.
Bounce. Lines play from circled Takes between their Take Line marks, so the Bounce's marks come for free. Lines with no circled Take carry forward the current Performance's slice and its coverage status. With no Performance, they get silence at the Line's duration (override, else estimate), with a warning that lists them. performance_source records the Takes used and any carried-forward Performance.
Import auto-placement. ffmpeg silencedetect (from the sidecar) finds speech islands. They are assigned in order to dialogue Lines, merged or split in proportion to the text estimates, and then corrected by hand. Forced alignment or ASR is a possible later strategy behind the same lane (out of scope).
Partition. Marks are contiguous, one boundary per Line plus an end mark, and no time is left unowned.
- A pause between consecutive dialogue Lines belongs to the earlier Line, so the boundary sits at the next speech start.
- Action Lines between dialogue split the silence between speech islands in proportion to their text estimates. Leading and trailing action Lines take the head and tail of the file.
- A Line shorter than the 1 s action minimum gets a warning badge and is not blocked.
- Transitions and shot directions are zero-length.
Dual dialogue. A simultaneous group has one span and is one unit for auto-placement. Its estimate is its longest member's.
Mutability. The audio is immutable, and new audio means a new Performance version. Marks are editable, undoable authored edits on a version: performance_mark(performance, line, start) plus an end. A Take records the Performance version and the slice it was sent (start and end). A mark edit therefore makes stale only Shots whose slice changed, with the Stale reason "Performance marks: Lines … moved". New audio is still the coarse "Performance version" reason.
Replacement. A new version gets fresh marks from its own source (a Bounce supplies them, an import runs auto-placement), with no carry-over. A ghost lane shows the previous version's marks for reference. Old versions keep their marks as history. Turning performance_off on or off leaves marks untouched.
Take Line marks. When a Take is queued, the Performance marks inside its slice are snapshotted into the job and become the Take's default Line marks, which can be nudged in Dailies as before. Later Performance mark edits never move them.
Coverage. A Performance covers a set of Lines. An import covers one contiguous run that the user picks (default: all Lines). A Bounce may have holes where it fell back to silence, and carry-forward keeps a hole a hole.
- Uncovered Lines take their duration from override, then estimate.
- A Shot entirely over uncovered Lines renders as if there were no Performance (H3 generates the audio).
- A Shot that straddles covered and uncovered Lines blocks queuing: "split the Shot or extend the Performance".
UI. A Performance lane on the Animatic surface, under the Line lane: the current version's waveform, a draggable boundary per Line, warning badges, an uncovered-Line shading, and the optional ghost lane. Its header has Import, Bounce, Re-run auto-placement (an undoable edit that replaces all marks) and the version history.

Effects on other decisions

".studio schema" (#11): adds performance_mark, a coverage flag per Line per version, a performance_source kind for carried-forward Performances, and the Take's recorded slice plus snapshotted marks.
"Staleness rules" (#12): adds a "Performance marks" Stale reason in the Performance group, judged by the slice diff.
Glossary: Performance and Line mark updated, Bounce added.

## Resolution 1. **Sources.** A Performance is either a **Bounce** of the Scene's Cut audio or **imported** audio. A source either supplies its own Line marks (a Bounce; a future per-Line TTS render) or needs marking (an import). Alternate voice sources such as local TTS are out of scope, but `performance_source` must stay open to new source kinds. 2. **Bounce.** Lines play from circled Takes between their Take Line marks, so the Bounce's marks come for free. Lines with no circled Take **carry forward** the current Performance's slice and its coverage status. With no Performance, they get silence at the Line's duration (override, else estimate), with a warning that lists them. `performance_source` records the Takes used and any carried-forward Performance. 3. **Import auto-placement.** ffmpeg `silencedetect` (from the sidecar) finds speech islands. They are assigned in order to dialogue Lines, merged or split in proportion to the text estimates, and then corrected by hand. Forced alignment or ASR is a possible later strategy behind the same lane (out of scope). 4. **Partition.** Marks are contiguous, one boundary per Line plus an end mark, and no time is left unowned. - A pause between consecutive dialogue Lines belongs to the earlier Line, so the boundary sits at the next speech start. - Action Lines between dialogue split the silence between speech islands in proportion to their text estimates. Leading and trailing action Lines take the head and tail of the file. - A Line shorter than the 1 s action minimum gets a warning badge and is not blocked. - Transitions and shot directions are zero-length. 5. **Dual dialogue.** A simultaneous group has one span and is one unit for auto-placement. Its estimate is its longest member's. 6. **Mutability.** The audio is immutable, and new audio means a new Performance version. Marks are editable, undoable authored edits on a version: `performance_mark(performance, line, start)` plus an end. A Take records the Performance version **and the slice it was sent** (start and end). A mark edit therefore makes stale only Shots whose slice changed, with the Stale reason "Performance marks: Lines … moved". New audio is still the coarse "Performance version" reason. 7. **Replacement.** A new version gets fresh marks from its own source (a Bounce supplies them, an import runs auto-placement), with no carry-over. A **ghost lane** shows the previous version's marks for reference. Old versions keep their marks as history. Turning `performance_off` on or off leaves marks untouched. 8. **Take Line marks.** When a Take is queued, the Performance marks inside its slice are snapshotted into the job and become the Take's default Line marks, which can be nudged in Dailies as before. Later Performance mark edits never move them. 9. **Coverage.** A Performance covers a **set** of Lines. An import covers one contiguous run that the user picks (default: all Lines). A Bounce may have holes where it fell back to silence, and carry-forward keeps a hole a hole. - Uncovered Lines take their duration from override, then estimate. - A Shot entirely over uncovered Lines renders as if there were no Performance (H3 generates the audio). - A Shot that **straddles** covered and uncovered Lines blocks queuing: "split the Shot or extend the Performance". 10. **UI.** A Performance lane on the Animatic surface, under the Line lane: the current version's waveform, a draggable boundary per Line, warning badges, an uncovered-Line shading, and the optional ghost lane. Its header has Import, Bounce, Re-run auto-placement (an undoable edit that replaces all marks) and the version history. **Effects on other decisions** - ".studio schema" (#11): adds `performance_mark`, a coverage flag per Line per version, a `performance_source` kind for carried-forward Performances, and the Take's recorded slice plus snapshotted marks. - "Staleness rules" (#12): adds a "Performance marks" Stale reason in the Performance group, judged by the slice diff. - Glossary: **Performance** and **Line mark** updated, **Bounce** added.

crowmaki closed this issue

2026-10-07 07:14:03 +00:00

crowmaki referenced this issue

2026-10-07 07:24:15 +00:00

Spec: AI Studio end-to-end slice #24

crowmaki referenced this issue

2026-10-07 07:34:52 +00:00

Performance import and marking #63

crowmaki referenced this issue

2026-10-09 09:53:47 +00:00

Performance import and marking (#63) #120