Queue a draft Take (#51) #105

Merged

crowmaki merged 3 commits from issue-51-queue-draft-take into main

2026-10-08 23:13:52 +00:00

crowmaki commented

2026-10-08 23:12:29 +00:00

Owner

Closes #51.

What changed

Queuing a Take (studio-core/src/takes.rs): queue_take(shot, TakeRequest { profile, seed }) builds the whole h3.draft job at queue time. The Take gets these at once:
- the Shot's next number (shot.next_take_number, never reused), named 12A-3
- the given seed or a random one below 2^53
- a copy of the Project's Generation profile, or the one from the queue dialog
Provenance: queuing also records what the Take was built from.
- One take_input row per reference sent. Each row has its slot number, medium, the exact bytes sent (as a blob) and real foreign keys to what it came from: the Lock snapshot, Keyframe, Tile, Character, Look, Wardrobe item, Prop or Location.
- The Shot's action text, its summary and the Setup's framing note, stored by value.
- On the take row: the prompt text and the filled graph as blobs, the overridden flag, the workflow id/version/hash, the ratio, the W×H, the requested ms and the frame count.
- take(id) returns all of this as TakeDetail.
Job builder: it fills the bindings, then makes the two structural edits:
- one LoadImage/LoadAudio per filled slot, wired into the MiniMaxH3ReferenceToVideo Autogrow ports in slot order
- a LoraLoaderModelOnly chain (enabled LoRAs only) on the manifest's chain point, with the shipped DMAD LoRA rewired to the chain's end
Sizes take the short side from the profile and the long side from the effective ratio, both rounded to ×32. The speed mode's step and strength values are applied. Each reference is uploaded under a name taken from its content hash, so the stored graph already names the exact files.
Slot media:
- Atlas and packed Prop come from the existing builders.
- A location angle is its snapshot pick.
- A Keyframe is its image.
- A Board is one image per Panel, or the Panels tiled onto one image on mid-grey.
- A Wardrobe mannequin is its snapshot Tiles packed side by side, like a Prop.
- A Voice reference with several clips is joined with ffmpeg into one FLAC, with 0.3 s gaps.
Rendition: the rendition row exists from queue time.
- On collect, the worker copies the MP4, the FLAC and both split latents to renditions/<id>.{mp4,flac} and latents/<id>.{video,audio}.latent.
- It keeps each latent's ComfyUI name (… [output]) for Promotion.
- It extracts a poster frame with ffmpeg.
- A draft whose job left out any of these files fails.
Worker:
- It runs preflight with the job's own profile.
- It uploads the job's inputs (job_upload) with overwrite=true before it submits.
- It records each output's ComfyUI file name.
ffmpeg (studio-core/src/ffmpeg.rs): only Rust calls it. It makes posters and joins audio. It is found through AI_STUDIO_FFMPEG, then next to the app executable (where Tauri puts sidecars), then on PATH.
studio: scheme: rendition/<id> serves a ready Rendition's video from the media folder, with Accept-Ranges and Range/206. That covers open-ended (capped at 4 MB a response), suffix and 416 cases. The Project lock is released before the file is read.
Schema V23:
- new tables take, take_input, rendition and job_upload
- shot.next_take_number
- keyframe.deleted_at
- the job table rebuilt with a rendition_id target
ADR-0004: is_consumed is now real.
- A Keyframe that a Take consumed is hidden instead of deleted when it is removed or replaced, and undo restores it.
- Shots, Setups and Entities that a Take consumed refuse deletion, through the existing "used by a Take" refusal.
UI: each Shot lists its Takes with a poster and a Queued / Rendering % / Ready / Failed badge. A ready Take plays its draft inline.
- Queue draft… opens a dialog with a seed field and the Generation profile editor. Changes there apply to that Take only.
- The editor is now ProfileEditor.svelte, shared with Project settings.
- Dialog gained a wide size.

Screenshots

The queue dialog. A LoRA added for this Take only, so the profile is marked Changed, and a typed seed:

The Shot's Takes after rendering: two ready drafts with posters and a failed one. Its tooltip says "ComfyUI failed in UNETLoader: CUDA out of memory". 1A-1 plays inline from studio://…/rendition/<id>, and seeking works through Range requests. The seed typed in the dialog (42) is the one recorded:

These were taken in the running app against a stand-in ComfyUI (the test fake, serving an ffmpeg test pattern as the H3 output).

Testing

cargo test --workspace --all-features
bun test src scripts
bun run check
New tests:
- crates/studio-core/tests/takes.rs checks naming and numbering, the recorded inputs with slots and foreign keys, and the filled graph: loader wiring by content name, the LoRA chain order, size, steps and seed. Against the fake ComfyUI it also checks that the graph sent equals the recorded one, the uploads, the Rendition files in the media folder, the poster, failed renders, and a profile LoRA ComfyUI lacks. It also covers consumed-Keyframe soft delete with undo, the Shot/Setup delete refusal, and refusals that use no number (a bad profile, a seed ≥ 2^53).
- crates/studio-core/src/ffmpeg.rs unit tests run real ffmpeg for the poster and the join.
- src-tauri/tests/takes.rs queues through the adapter, renders against the fake, and checks 200/206/416 Range serving.
- src/lib/scenes/takes.test.ts covers the badges, the seed parsing and the profile override.

Review notes

The ffmpeg sidecar isn't bundled yet. Adding externalBin to tauri.conf.json would break tauri dev and builds for anyone without the binary in src-tauri/binaries/. So this PR finds ffmpeg at runtime instead: the env var, then next to the exe, then PATH. Bundling an LGPL build (and a fetch script) can follow once we pick a build. Tests need ffmpeg on PATH or AI_STUDIO_FFMPEG.
Missing poster: if ffmpeg can't make a poster, the Rendition still becomes ready without one, and the reason is logged. The "proxy missing"-style marker and retry belong to the post-step lane in #54.
Profile file missing: if a profile file is missing in ComfyUI, the job fails, as image jobs already do. Making it wait instead is #53.
My own choices, not spelled out in the spec:
- the mannequin image (snapshot Tiles side by side)
- the tiled-Board layout (a near-square grid in reading order)
- the 0.3 s gap between joined voice clips
Board annotations aren't flattened: Boards are sent as their Sketch layer only, as board_images already does. Flattening the annotation layer in Rust is still to do.
Text inputs are recorded by value only. The spec allows value or hash, and the fields are short.

Closes #51. ## What changed - **Queuing a Take** (`studio-core/src/takes.rs`): `queue_take(shot, TakeRequest { profile, seed })` builds the whole `h3.draft` job at queue time. The Take gets these at once: - the Shot's next number (`shot.next_take_number`, never reused), named `12A-3` - the given seed or a random one below 2^53 - a copy of the Project's Generation profile, or the one from the queue dialog - **Provenance**: queuing also records what the Take was built from. - One `take_input` row per reference sent. Each row has its slot number, medium, the exact bytes sent (as a blob) and real foreign keys to what it came from: the Lock snapshot, Keyframe, Tile, Character, Look, Wardrobe item, Prop or Location. - The Shot's action text, its summary and the Setup's framing note, stored by value. - On the `take` row: the prompt text and the filled graph as blobs, the overridden flag, the workflow id/version/hash, the ratio, the W×H, the requested ms and the frame count. - `take(id)` returns all of this as `TakeDetail`. - **Job builder**: it fills the bindings, then makes the two structural edits: - one `LoadImage`/`LoadAudio` per filled slot, wired into the `MiniMaxH3ReferenceToVideo` Autogrow ports in slot order - a `LoraLoaderModelOnly` chain (enabled LoRAs only) on the manifest's chain point, with the shipped DMAD LoRA rewired to the chain's end Sizes take the short side from the profile and the long side from the effective ratio, both rounded to ×32. The speed mode's step and strength values are applied. Each reference is uploaded under a name taken from its content hash, so the stored graph already names the exact files. - **Slot media**: - Atlas and packed Prop come from the existing builders. - A location angle is its snapshot pick. - A Keyframe is its image. - A Board is one image per Panel, or the Panels tiled onto one image on mid-grey. - A Wardrobe mannequin is its snapshot Tiles packed side by side, like a Prop. - A Voice reference with several clips is joined with ffmpeg into one FLAC, with 0.3 s gaps. - **Rendition**: the `rendition` row exists from queue time. - On collect, the worker copies the MP4, the FLAC and both split latents to `renditions/<id>.{mp4,flac}` and `latents/<id>.{video,audio}.latent`. - It keeps each latent's ComfyUI name (`… [output]`) for Promotion. - It extracts a poster frame with ffmpeg. - A draft whose job left out any of these files fails. - **Worker**: - It runs preflight with the job's own profile. - It uploads the job's inputs (`job_upload`) with `overwrite=true` before it submits. - It records each output's ComfyUI file name. - **ffmpeg** (`studio-core/src/ffmpeg.rs`): only Rust calls it. It makes posters and joins audio. It is found through `AI_STUDIO_FFMPEG`, then next to the app executable (where Tauri puts sidecars), then on `PATH`. - **`studio:` scheme**: `rendition/<id>` serves a ready Rendition's video from the media folder, with `Accept-Ranges` and Range/206. That covers open-ended (capped at 4 MB a response), suffix and 416 cases. The Project lock is released before the file is read. - **Schema V23**: - new tables `take`, `take_input`, `rendition` and `job_upload` - `shot.next_take_number` - `keyframe.deleted_at` - the `job` table rebuilt with a `rendition_id` target - **ADR-0004**: `is_consumed` is now real. - A Keyframe that a Take consumed is hidden instead of deleted when it is removed or replaced, and undo restores it. - Shots, Setups and Entities that a Take consumed refuse deletion, through the existing "used by a Take" refusal. - **UI**: each Shot lists its Takes with a poster and a Queued / Rendering % / Ready / Failed badge. A ready Take plays its draft inline. - **Queue draft…** opens a dialog with a seed field and the Generation profile editor. Changes there apply to that Take only. - The editor is now `ProfileEditor.svelte`, shared with Project settings. - `Dialog` gained a `wide` size. ## Screenshots The queue dialog. A LoRA added for this Take only, so the profile is marked Changed, and a typed seed: ![Queue dialog](https://git.crowmaki.me/attachments/c846ef9a-b627-46b4-b6ec-c0c7c94192cf) The Shot's Takes after rendering: two ready drafts with posters and a failed one. Its tooltip says "ComfyUI failed in UNETLoader: CUDA out of memory". `1A-1` plays inline from `studio://…/rendition/<id>`, and seeking works through Range requests. The seed typed in the dialog (42) is the one recorded: ![Take list and draft player](https://git.crowmaki.me/attachments/ba17f39a-f4c5-4049-8ee5-92f74b6358dc) These were taken in the running app against a stand-in ComfyUI (the test fake, serving an ffmpeg test pattern as the H3 output). ## Testing - `cargo test --workspace --all-features` - `bun test src scripts` - `bun run check` - New tests: - `crates/studio-core/tests/takes.rs` checks naming and numbering, the recorded inputs with slots and foreign keys, and the filled graph: loader wiring by content name, the LoRA chain order, size, steps and seed. Against the fake ComfyUI it also checks that the graph sent equals the recorded one, the uploads, the Rendition files in the media folder, the poster, failed renders, and a profile LoRA ComfyUI lacks. It also covers consumed-Keyframe soft delete with undo, the Shot/Setup delete refusal, and refusals that use no number (a bad profile, a seed ≥ 2^53). - `crates/studio-core/src/ffmpeg.rs` unit tests run real ffmpeg for the poster and the join. - `src-tauri/tests/takes.rs` queues through the adapter, renders against the fake, and checks 200/206/416 Range serving. - `src/lib/scenes/takes.test.ts` covers the badges, the seed parsing and the profile override. ## Review notes - **The ffmpeg sidecar isn't bundled yet.** Adding `externalBin` to `tauri.conf.json` would break `tauri dev` and builds for anyone without the binary in `src-tauri/binaries/`. So this PR finds ffmpeg at runtime instead: the env var, then next to the exe, then `PATH`. Bundling an LGPL build (and a fetch script) can follow once we pick a build. Tests need `ffmpeg` on `PATH` or `AI_STUDIO_FFMPEG`. - **Missing poster**: if ffmpeg can't make a poster, the Rendition still becomes ready without one, and the reason is logged. The "proxy missing"-style marker and retry belong to the post-step lane in #54. - **Profile file missing**: if a profile file is missing in ComfyUI, the job fails, as image jobs already do. Making it wait instead is #53. - **My own choices**, not spelled out in the spec: - the mannequin image (snapshot Tiles side by side) - the tiled-Board layout (a near-square grid in reading order) - the 0.3 s gap between joined voice clips - **Board annotations aren't flattened**: Boards are sent as their Sketch layer only, as `board_images` already does. Flattening the annotation layer in Rust is still to do. - **Text inputs** are recorded by value only. The spec allows value or hash, and the fields are short.

51-queue-dialog.png

74 KiB

51-take-list.png

89 KiB

crowmaki added 3 commits

2026-10-08 23:12:29 +00:00

Queue a draft Take and collect its Rendition (#51 ) a8a8b5347c

A Take is built at queue time from the Shot's current inputs: next
number, seed, copied Generation profile, take_input provenance rows with
real foreign keys and the exact bytes sent, prompt and graph blobs, and
the shipped workflow id/version/hash. The job builder wires one loader
per filled slot and the profile's LoRA chain. The worker uploads the
content-named inputs, checks the profile in preflight, and collects the
MP4, FLAC and both latents into the media folder with an ffmpeg poster.
The studio: scheme serves rendition/<id> with Range/206.

Keyframes a Take consumed are hidden rather than deleted (ADR-0004);
Shots and Setups with Takes refuse deletion.

Take list, queue dialog and draft player on Shots (#51 ) 5e808d3aa8

Each Shot lists its Takes with Queued / Rendering % / Ready / Failed
badges and posters; a ready Take plays its draft from the studio:
scheme. Queue draft... opens a dialog that copies the Project's
Generation profile for one Take and takes an optional seed. The profile
editor is shared with Project settings.

Address review of draft Take queuing (#51 ) b86ee6e88b

Refuse a seed of 2^53 or more instead of masking it, fail a draft
whose job left out its audio or a latent, log a poster that could not
be made, and name the queue dialog's helpers for what they return.

crowmaki merged commit de5f243e21 into main

2026-10-08 23:13:52 +00:00

crowmaki deleted branch issue-51-queue-draft-take

2026-10-08 23:13:53 +00:00

crowmaki referenced this pull request from a commit

2026-10-08 23:13:54 +00:00

Merge pull request 'Queue a draft Take (#51)' (#105) from issue-51-queue-draft-take into main