Shipped workflow roster and slot contracts

crowmaki commented

2026-10-06 03:55:56 +00:00

Owner

Part of #1
Blocked by: #13, #14, #15, #9, #20

Question

Which ComfyUI workflows does the app ship (reference t2i, board t2i, H3 draft, H3 Promotion, H3 single-pass final…)? What are each one's named inputs and outputs, model/LoRA selection points and versioning, and what does preflight check for each?

Part of #1 Blocked by: #13, #14, #15, #9, #20 ## Question Which ComfyUI workflows does the app ship (reference t2i, board t2i, H3 draft, H3 Promotion, H3 single-pass final…)? What are each one's named inputs and outputs, model/LoRA selection points and versioning, and what does preflight check for each?

crowmaki added the

wayfinder:grilling

label

2026-10-06 03:55:56 +00:00

crowmaki referenced this issue

2026-10-06 03:56:04 +00:00

Render worker scheduling #18

crowmaki referenced this issue

2026-10-06 03:56:08 +00:00

Wayfinder: AI Studio end-to-end slice spec #1

crowmaki commented

2026-10-06 05:08:56 +00:00

Author

Owner

Input from Rust core vs TypeScript frontend split: Renditions are played in WebView2, which only plays H.264/AAC (no ProRes). Either the shipped workflows encode playable Renditions, or the bundled LGPL ffmpeg sidecar makes proxies. ffmpeg is never used to encode Renditions, which stays in ComfyUI.

Input from [Rust core vs TypeScript frontend split](https://git.crowmaki.me/crowmaki/ai-studio/issues/3): Renditions are played in WebView2, which only plays H.264/AAC (no ProRes). Either the shipped workflows encode playable Renditions, or the bundled LGPL ffmpeg sidecar makes proxies. ffmpeg is never used to encode Renditions, which stays in ComfyUI.

crowmaki referenced this issue

2026-10-06 06:06:20 +00:00

Board paint tool and Notation #6

crowmaki commented

2026-10-06 06:07:50 +00:00

Author

Owner

From "Board paint tool and Notation" (#6): reducing t2i output to 3 tones is opt-in, so the board t2i workflow's output is used as generated. Its prompting therefore has to keep Boards rough and non-photoreal by itself. Boards are sent at native canvas size (short side 180 px, Project aspect ratio), with no upscaling.

crowmaki commented

2026-10-06 09:11:15 +00:00

Author

Owner

From "Verify latent round trip and refine-pass cost" (#20):

The draft/Promotion split is verified. Draft = first pass → LTXVSeparateAVLatent → 2× SaveLatent. Promotion = 2× LoadLatent "<subfolder>/<file> [output]" → upscaler → concat → 2nd pass, with conditioning rebuilt. Read the saved file names from history outputs[<node>].latents.
Audio decision needed: with generated audio, the 2nd pass re-noises the audio (~15 dB SNR run to run), so a promoted final's audio differs from the circled draft's audio. If Promotion must keep the draft's audio (for example when that audio becomes the Performance), the Promotion workflow must mask the loaded audio latent with a 0 SetLatentNoiseMask every time, not only for forced audio.
ResolutionSelector 1.0 MP / 16:9 / ×32 gives 1376×768, not 1344×768. The roster should pin explicit draft dimensions.
At 1984×1120, 15 s, the Promotion takes ~283 s and fits in 32 GB.

From "Verify latent round trip and refine-pass cost" (#20): - The draft/Promotion split is verified. Draft = first pass → `LTXVSeparateAVLatent` → 2× `SaveLatent`. Promotion = 2× `LoadLatent "<subfolder>/<file> [output]"` → upscaler → concat → 2nd pass, with conditioning rebuilt. Read the saved file names from history `outputs[<node>].latents`. - **Audio decision needed:** with generated audio, the 2nd pass re-noises the audio (~15 dB SNR run to run), so a promoted final's audio differs from the circled draft's audio. If Promotion must keep the draft's audio (for example when that audio becomes the Performance), the Promotion workflow must mask the loaded audio latent with a 0 `SetLatentNoiseMask` every time, not only for forced audio. - ResolutionSelector 1.0 MP / 16:9 / ×32 gives 1376×768, not 1344×768. The roster should pin explicit draft dimensions. - At 1984×1120, 15 s, the Promotion takes ~283 s and fits in 32 GB.

crowmaki commented

2026-10-07 04:06:04 +00:00

Author

Owner

From "Reference slot budget" (#10), slot contracts: at most 9 image + 3 audio slots are used, and video slots are unused in the slice. Derived inputs the job builder must produce: Atlases (already specified), a packed Prop image (the Prop Lock snapshot's Tiles side by side in uniform cells on mid-grey, identified by content), a tiled Board (or one slot per Panel), and a joined Voice reference (a Character's snapshot clips concatenated with short gaps into one audio slot).

From "Reference slot budget" (#10), slot contracts: at most 9 image + 3 audio slots are used, and video slots are unused in the slice. Derived inputs the job builder must produce: Atlases (already specified), a **packed Prop image** (the Prop Lock snapshot's Tiles side by side in uniform cells on mid-grey, identified by content), a tiled Board (or one slot per Panel), and a **joined Voice reference** (a Character's snapshot clips concatenated with short gaps into one audio slot).

crowmaki self-assigned this

2026-10-07 05:01:51 +00:00

crowmaki commented

2026-10-07 06:12:16 +00:00

Author

Owner

Resolution

Roster: five shipped workflows.
- h3.draft: first pass at draft size. Writes an H.264 MP4, a lossless WAV and the split AV latent (LTXVSeparateAVLatent → 2× SaveLatent).
- h3.promotion: 2× LoadLatent … [output] → latent upscaler 3D → concat → refine pass at final size, with the conditioning rebuilt.
- h3.final (Single-pass final): generates directly at final size in one pass, with no draft, no upscaler and no refine. It looks different from (and usually better than) a promoted draft with the same seed. It does not save a latent.
- image.t2i: reference Tiles, Wardrobe mannequins, location angles, Prop Tiles and Keyframes.
- image.board: Boards. It is a separate workflow with its own fixed style prompt, so that it can drift apart from image.t2i.
- Both image workflows target Qwen-Image 2.1.
- Not in the slice: SeedVR2, RIFE (opt-in, never part of a final), and Keyframe-from-Take-frame (a Rust/ffmpeg frame grab, not ComfyUI).
Generation profile (per Take; the Project default is copied).
- Pickable from /api/models/*:
  - the diffusion model file: FL2VA or Ref2VA, any quantization.
  - the text encoder file.
  - a user LoRA stack (file + weight, applied to both passes).
  - a speed mode, Distilled or Full. Distilled applies the DMAD 4-step LoRA with the shipped step and scheduler values. Full uses no distill LoRA and the shipped higher step counts. The distill LoRA is never an entry in the user stack, and steps are never typed in by hand.
- Sizes: a short side per quality, with defaults 768 (draft) and 1120 (final). The long side follows the Shot's effective aspect ratio, and both sides are rounded to ×32. Rust computes them, so there is no ResolutionSelector in shipped graphs. Promotion and single-pass finals use the same final size.
- Fixed in the workflow (a change means a new version): VAEs (video minimax_h3_video_vae_int8_convrot, audio fp32), upscaler weights, sampler and schedulers, refine denoise, BlockSparseAttention and the low-VRAM settings.
Storage and filling: template + manifest + two splice points.
- Each workflow is an API-format JSON file in the repo, with a manifest that names its bindings:
  - scalar inputs (prompt, seed, width, height, length, model files…) as node_id.input.
  - named output nodes.
  - for each non-core class_type, the custom node pack name and repo URL.
  - for each fixed model file, its folder and a download URL.
- Rust makes only two structural edits:
  - it adds one LoadImage/LoadAudio per filled slot and wires each into the MiniMaxH3ReferenceToVideo Autogrow ports in slot order.
  - it splices a chain of LoraLoaderModelOnly nodes at a named model point.
- The filled-in graph is the job blob.
- Unit tests check that every manifest binding resolves against its template.
Versioning.
- Each workflow has a stable id and a hand-bumped integer version. Bump it when the output would change.
- CI fails if the content hash changes without a bump.
- Every job row records the id, version and content hash. These feed the Workflow Stale reason, which stays informational only.
- Promotion always uses the current h3.promotion. The saved latent is the contract between the two jobs.
- Old template versions are not kept, because job blobs can be resubmitted.
Preflight.
- Runs on connect, on demand from a ComfyUI status panel, and before each submission (for that workflow and that job's profile files). ComfyUI validation is the backstop.
- Checks per workflow:
  - ComfyUI is reachable and at version ≥ 0.39.0 (/system_stats).
  - Every class_type is present. A missing one is reported as the node pack name and URL.
  - Every fixed model file is in its loader combo (reported with folder and URL).
  - Every fixed combo value is still valid.
- Profile files are checked per Take at queue time, and the default profile on connect.
- Each workflow is Ready or Not ready (reasons). Queue actions for a Not-ready workflow are disabled, and its already-queued jobs wait rather than fail. Preflight never installs or downloads anything.
Outputs.
- Drafts: an H.264/AAC MP4 at 24 fps, playable in-app.
- Finals: a ProRes 422 HQ .mov master written by the workflow, plus an H.264 proxy made by the LGPL ffmpeg sidecar (NVENC/OpenH264) for in-app playback.
- Every H3 job also writes lossless WAV audio (a Performance from a Take's audio uses it).
- Image workflows: PNG through SaveImage, pulled through /view and stored as blobs.
Image sizes (set by the caller, in app code):
- turnaround Tiles and Wardrobe mannequins: 4:9 at ~1 MP (the Atlas cell shape; the face close-up uses the same cell).
- location angles: Project aspect ratio at ~1 MP.
- Prop Tiles: 1:1 at ~1 MP.
- Keyframes: exactly the Shot's final W×H.
- Boards: ~1 MP at the Panel's aspect ratio, downscaled by Rust to the Panel canvas.
Audio.
- h3.promotion always freezes the draft's audio with a 0 SetLatentNoiseMask, with no toggle. A promoted final's audio equals its draft's, so Line marks stay valid.
- With a Performance, Rust cuts the Shot's slice (Lines plus handles), pads it to the job's frame count, and sends it both as forced latent audio (VAEEncodeAudio → 0 mask) and as the fully_copy audio slot.
Length. Seconds round up to the H3 grid of 5 + 17k frames (≤ 362). The extra frames are tail handle. The job records both the requested seconds and the actual frame count.
Re-promotion.
- The refine noise seed belongs to each Promotion: a new random seed by default, editable in the queue dialog, and recorded on the final Rendition.
- Re-promoting replaces the Take's final in place once the new one succeeds. A failed or cancelled re-promotion leaves the old final. While it runs, Dailies shows the old final with a "re-promoting" badge.
- The Take goes back to unseen, while its circles, pin, notes and Line marks stay.
- Only promoted finals can be re-promoted. Redoing a single-pass final means a new Take.

Glossary: added Single-pass final and Generation profile, and extended Promotion (frozen audio, re-promotion in place).

Out of scope: SeedVR2 and NVIDIA upscaling as later drop-in finishing plugins.

## Resolution 1. **Roster: five shipped workflows.** - `h3.draft`: first pass at draft size. Writes an H.264 MP4, a lossless WAV and the split AV latent (`LTXVSeparateAVLatent` → 2× `SaveLatent`). - `h3.promotion`: 2× `LoadLatent … [output]` → latent upscaler 3D → concat → refine pass at final size, with the conditioning rebuilt. - `h3.final` (**Single-pass final**): generates directly at final size in one pass, with no draft, no upscaler and no refine. It looks different from (and usually better than) a promoted draft with the same seed. It does not save a latent. - `image.t2i`: reference Tiles, Wardrobe mannequins, location angles, Prop Tiles and Keyframes. - `image.board`: Boards. It is a separate workflow with its own fixed style prompt, so that it can drift apart from `image.t2i`. - Both image workflows target **Qwen-Image 2.1**. - Not in the slice: SeedVR2, RIFE (opt-in, never part of a final), and Keyframe-from-Take-frame (a Rust/ffmpeg frame grab, not ComfyUI). 2. **Generation profile (per Take; the Project default is copied).** - Pickable from `/api/models/*`: - the diffusion model file: FL2VA or Ref2VA, any quantization. - the text encoder file. - a user LoRA stack (file + weight, applied to both passes). - a **speed mode**, *Distilled* or *Full*. *Distilled* applies the DMAD 4-step LoRA with the shipped step and scheduler values. *Full* uses no distill LoRA and the shipped higher step counts. The distill LoRA is never an entry in the user stack, and steps are never typed in by hand. - **Sizes:** a short side per quality, with defaults **768** (draft) and **1120** (final). The long side follows the Shot's effective aspect ratio, and both sides are rounded to ×32. Rust computes them, so there is no `ResolutionSelector` in shipped graphs. Promotion and single-pass finals use the same final size. - **Fixed in the workflow** (a change means a new version): VAEs (video `minimax_h3_video_vae_int8_convrot`, audio fp32), upscaler weights, sampler and schedulers, refine denoise, BlockSparseAttention and the low-VRAM settings. 3. **Storage and filling: template + manifest + two splice points.** - Each workflow is an API-format JSON file in the repo, with a manifest that names its bindings: - scalar inputs (`prompt`, `seed`, `width`, `height`, `length`, model files…) as `node_id.input`. - named output nodes. - for each non-core class_type, the custom node pack name and repo URL. - for each fixed model file, its folder and a download URL. - Rust makes only two structural edits: - it adds one `LoadImage`/`LoadAudio` per filled slot and wires each into the `MiniMaxH3ReferenceToVideo` Autogrow ports in slot order. - it splices a chain of `LoraLoaderModelOnly` nodes at a named model point. - The filled-in graph is the job blob. - Unit tests check that every manifest binding resolves against its template. 4. **Versioning.** - Each workflow has a stable id and a hand-bumped integer version. Bump it when the output would change. - CI fails if the content hash changes without a bump. - Every `job` row records the id, version and content hash. These feed the Workflow Stale reason, which stays informational only. - Promotion always uses the **current** `h3.promotion`. The saved latent is the contract between the two jobs. - Old template versions are not kept, because job blobs can be resubmitted. 5. **Preflight.** - Runs on connect, on demand from a ComfyUI status panel, and before each submission (for that workflow and that job's profile files). ComfyUI validation is the backstop. - Checks per workflow: - ComfyUI is reachable and at version ≥ 0.39.0 (`/system_stats`). - Every class_type is present. A missing one is reported as the node pack name and URL. - Every fixed model file is in its loader combo (reported with folder and URL). - Every fixed combo value is still valid. - Profile files are checked per Take at queue time, and the default profile on connect. - Each workflow is *Ready* or *Not ready (reasons)*. Queue actions for a Not-ready workflow are disabled, and its already-queued jobs **wait** rather than fail. Preflight never installs or downloads anything. 6. **Outputs.** - Drafts: an H.264/AAC MP4 at 24 fps, playable in-app. - Finals: a **ProRes 422 HQ `.mov` master** written by the workflow, plus an H.264 **proxy** made by the LGPL ffmpeg sidecar (NVENC/OpenH264) for in-app playback. - Every H3 job also writes **lossless WAV** audio (a Performance from a Take's audio uses it). - Image workflows: PNG through `SaveImage`, pulled through `/view` and stored as blobs. 7. **Image sizes (set by the caller, in app code):** - turnaround Tiles and Wardrobe mannequins: 4:9 at ~1 MP (the Atlas cell shape; the face close-up uses the same cell). - location angles: Project aspect ratio at ~1 MP. - Prop Tiles: 1:1 at ~1 MP. - Keyframes: exactly the Shot's final W×H. - Boards: ~1 MP at the Panel's aspect ratio, downscaled by Rust to the Panel canvas. 8. **Audio.** - `h3.promotion` **always** freezes the draft's audio with a 0 `SetLatentNoiseMask`, with no toggle. A promoted final's audio equals its draft's, so Line marks stay valid. - With a Performance, Rust cuts the Shot's slice (Lines plus handles), pads it to the job's frame count, and sends it **both** as forced latent audio (`VAEEncodeAudio` → 0 mask) and as the `fully_copy` audio slot. 9. **Length.** Seconds round **up** to the H3 grid of 5 + 17k frames (≤ 362). The extra frames are tail handle. The job records both the requested seconds and the actual frame count. 10. **Re-promotion.** - The refine noise seed belongs to each Promotion: a new random seed by default, editable in the queue dialog, and recorded on the final Rendition. - Re-promoting replaces the Take's final **in place** once the new one succeeds. A failed or cancelled re-promotion leaves the old final. While it runs, Dailies shows the old final with a "re-promoting" badge. - The Take goes back to unseen, while its circles, pin, notes and Line marks stay. - Only promoted finals can be re-promoted. Redoing a single-pass final means a new Take. Glossary: added **Single-pass final** and **Generation profile**, and extended **Promotion** (frozen audio, re-promotion in place). Out of scope: SeedVR2 and NVIDIA upscaling as later drop-in finishing plugins.

crowmaki closed this issue

2026-10-07 06:12:16 +00:00

crowmaki referenced this issue

2026-10-07 06:12:26 +00:00

Edit package layout #19

crowmaki referenced this issue

2026-10-07 06:12:26 +00:00

Render worker scheduling #18

crowmaki referenced this issue

2026-10-07 06:24:30 +00:00

Render worker scheduling #18

crowmaki referenced this issue