Keyframes: import and generate (#45) #96

Merged

crowmaki merged 2 commits from issue-45-keyframes into main

2026-10-08 19:09:30 +00:00

crowmaki commented

2026-10-08 18:51:52 +00:00

Owner

Closes #45.

A filmmaker can pin how a Shot opens, closes or looks at a given time (User Story 91). Keyframes taken from another Take's frame come in #60.

What changed

Schema v17. A new keyframe table holds the anchor (start, end, or at with a time in ms), the origin (imported or generated), the image blob, and the prompt and seed of a generated one. A Shot has at most one Keyframe per anchor. The job table is rebuilt so a job can target an anchor of a Shot.
import_keyframe stores the image with its pixels as they came. Putting a Keyframe at a taken anchor replaces it as one undoable edit. remove_keyframe is undoable. Shot and Setup deletes take their Keyframes with them, and undo brings them back.
generate_keyframe queues image.t2i at the Shot's exact final W×H: a 1120 px short side, with the long side following the effective ratio, both rounded to ×32 (16:9 gives 1984×1120). The image lands at its anchor as an undoable "Generate keyframe" edit.
Old ratio. off_ratio flags a Keyframe whose image is more than 1% away from the Shot's effective ratio. The Board view shows the flag as a badge.
Board view. A Keyframes strip under the stage for the active Shot. It shows Start and End and any times between, each with Import…, Generate…, remove, and the job's queued, running or failed state. A time field adds a Keyframe at a time.
Queue entries are labelled 1A · Start keyframe or 1A · Keyframe at 1.5 s. JobDto gains keyframeAnchor, and shotId is set for keyframe jobs.

Decisions worth a look

"Flagged at send time" is half done: the flag exists. Nothing sends Takes yet, so #51 has to read off_ratio when it builds the job. require_queueable_length already works the same way.
The final size is a stand-in. FINAL_SHORT_SIDE = 1120 is the Generation profile's default. #50 should replace it with the Project's profile value.
One Keyframe per anchor, replace on put. A second Keyframe at the same anchor replaces the first, and undo restores it. This keeps "how the Shot opens" unambiguous. Generating at an anchor whose job still waits or runs is refused, as Describe… does.
Milliseconds, not seconds. The spec sketches at_seconds?. I stored at_ms, which matches Shot lengths everywhere else.
Times are checked when added. A time must fall inside the Shot when the Keyframe is added. A Shot shortened later can leave a Keyframe past its end, and nothing flags that yet.
Split. Keyframes stay with the Shot, which is the first part after a split.
Jobs whose Shot was deleted. Like describe jobs, a keyframe job only points at its Shot and has no foreign key. While the Shot is gone, the job is hidden from the queue and does not run. Undo brings both back.
Landed size. The image ComfyUI returns is stored as it is. Its size is not checked against the size requested.

Tests

crates/studio-core/tests/keyframes.rs has 17 tests:
- final sizes for every ratio
- start, end and time anchors, and time order
- times outside the Shot
- replace and undo
- remove
- Shot and Setup deletes
- off_ratio under Project and Shot ratio changes
- the submitted image.t2i graph at the exact final size
- landing, replace and failure through the worker and the fake ComfyUI
- jobs whose Shot was deleted
A migration test checks that describe jobs survive the v17 rebuild and that the new CHECKs hold.
The adapter smoke test covers all four commands.
src/lib/board/keyframes.test.ts covers slot layout, labels and badges.
Full cargo test --workspace --all-features (487 passed), bun test src (99 passed) and bun run check pass. Clippy adds no new warnings.
Run in the real app against a live ComfyUI. A 16:9 import at Start, and a square import at End showing the Old ratio badge. Generate… at 1.5 s rendered a real 1984×1120 image, which landed with "Undo Generate keyframe".

Closes #45. A filmmaker can pin how a Shot opens, closes or looks at a given time (User Story 91). Keyframes taken from another Take's frame come in #60. ## What changed - **Schema v17.** A new `keyframe` table holds the anchor (`start`, `end`, or `at` with a time in ms), the origin (`imported` or `generated`), the image blob, and the prompt and seed of a generated one. A Shot has at most one Keyframe per anchor. The job table is rebuilt so a job can target an anchor of a Shot. - **`import_keyframe`** stores the image with its pixels as they came. Putting a Keyframe at a taken anchor replaces it as one undoable edit. **`remove_keyframe`** is undoable. Shot and Setup deletes take their Keyframes with them, and undo brings them back. - **`generate_keyframe`** queues `image.t2i` at the Shot's exact final W×H: a 1120 px short side, with the long side following the effective ratio, both rounded to ×32 (16:9 gives 1984×1120). The image lands at its anchor as an undoable "Generate keyframe" edit. - **Old ratio.** `off_ratio` flags a Keyframe whose image is more than 1% away from the Shot's effective ratio. The Board view shows the flag as a badge. - **Board view.** A Keyframes strip under the stage for the active Shot. It shows Start and End and any times between, each with Import…, Generate…, remove, and the job's queued, running or failed state. A time field adds a Keyframe at a time. - Queue entries are labelled `1A · Start keyframe` or `1A · Keyframe at 1.5 s`. `JobDto` gains `keyframeAnchor`, and `shotId` is set for keyframe jobs. ## Decisions worth a look - **"Flagged at send time"** is half done: the flag exists. Nothing sends Takes yet, so #51 has to read `off_ratio` when it builds the job. `require_queueable_length` already works the same way. - **The final size is a stand-in.** `FINAL_SHORT_SIDE = 1120` is the Generation profile's default. #50 should replace it with the Project's profile value. - **One Keyframe per anchor, replace on put.** A second Keyframe at the same anchor replaces the first, and undo restores it. This keeps "how the Shot opens" unambiguous. Generating at an anchor whose job still waits or runs is refused, as Describe… does. - **Milliseconds, not seconds.** The spec sketches `at_seconds?`. I stored `at_ms`, which matches Shot lengths everywhere else. - **Times are checked when added.** A time must fall inside the Shot when the Keyframe is added. A Shot shortened later can leave a Keyframe past its end, and nothing flags that yet. - **Split.** Keyframes stay with the Shot, which is the first part after a split. - **Jobs whose Shot was deleted.** Like describe jobs, a keyframe job only points at its Shot and has no foreign key. While the Shot is gone, the job is hidden from the queue and does not run. Undo brings both back. - **Landed size.** The image ComfyUI returns is stored as it is. Its size is not checked against the size requested. ## Tests - `crates/studio-core/tests/keyframes.rs` has 17 tests: - final sizes for every ratio - start, end and time anchors, and time order - times outside the Shot - replace and undo - remove - Shot and Setup deletes - `off_ratio` under Project and Shot ratio changes - the submitted `image.t2i` graph at the exact final size - landing, replace and failure through the worker and the fake ComfyUI - jobs whose Shot was deleted - A migration test checks that describe jobs survive the v17 rebuild and that the new CHECKs hold. - The adapter smoke test covers all four commands. - `src/lib/board/keyframes.test.ts` covers slot layout, labels and badges. - Full `cargo test --workspace --all-features` (487 passed), `bun test src` (99 passed) and `bun run check` pass. Clippy adds no new warnings. - **Run in the real app against a live ComfyUI.** A 16:9 import at Start, and a square import at End showing the Old ratio badge. Generate… at 1.5 s rendered a real 1984×1120 image, which landed with "Undo Generate keyframe".

crowmaki added 2 commits

2026-10-08 18:51:52 +00:00

Keyframes: import and generate (#45 ) 5c7b9ab8c4

A Shot can pin how it opens, closes or looks at a time inside it.

- Schema v17: a keyframe table (start, end or a time in ms; imported or
  generated, with the prompt and seed) with one Keyframe per anchor, and a
  job target for a Shot's anchor. Undo can bring a deleted Shot back, so a
  keyframe job only points at its Shot and is left out of the queue while
  the Shot is gone.
- import_keyframe keeps the image's pixels; putting a Keyframe at a taken
  anchor replaces it as one undoable edit. remove_keyframe is undoable.
  Shot and Setup deletes take their Keyframes along.
- generate_keyframe queues image.t2i at the Shot's exact final W x H: a
  1120 px short side, the long side following the effective ratio, both on
  a 32 px grid. The image lands at its anchor as an undoable edit.
- A Keyframe whose image is not at the Shot's effective ratio is flagged
  off_ratio, for queuing to report when it sends it.
- The Board view gets a Keyframes strip for the active Shot: start, end
  and any times between, with Import..., Generate..., remove, an Old ratio
  badge and the job's state.

Address review of Keyframes (#45 ) 17d8e37264

Removing a Keyframe checks that no Take used it, as Shot and Setup deletes
do. A job's change notice names the Shot rather than the Board, as it now
covers Keyframes too, and the Panel and Keyframe arms of finish_job share
one landing path. The 1120 px final short side is private and marked as the
Generation profile's default it stands in for.

crowmaki merged commit 1d0a64ad8d into main

2026-10-08 19:09:30 +00:00

crowmaki deleted branch issue-45-keyframes

2026-10-08 19:09:30 +00:00

crowmaki referenced this pull request from a commit

2026-10-08 19:09:32 +00:00

Merge pull request 'Keyframes: import and generate (#45)' (#96) from issue-45-keyframes into main