crowmaki/ai-studio

Watch

Fork 0

Reusing H3 AV latents across jobs #14

New issue

Closed

opened 2026-10-06 03:55:55 +00:00 by crowmaki · 1 comment

crowmaki commented

2026-10-06 03:55:55 +00:00

Owner

Part of #1

Question

Can a draft Take's H3 audio-video latent be saved by one ComfyUI job and loaded by a later job, so Promotion runs the upscale pass without regenerating? Which nodes handle this (SaveLatent/LoadLatent and similar, or the LTXV AV concat/separate nodes used in scratch/minimaxH3SEEDHUNTERLatent_v25.json)? Where are the files written, how big are they, and do they survive ComfyUI restarts?

Part of #1 ## Question Can a draft Take's H3 audio-video latent be saved by one ComfyUI job and loaded by a later job, so Promotion runs the upscale pass without regenerating? Which nodes handle this (SaveLatent/LoadLatent and similar, or the LTXV AV concat/separate nodes used in `scratch/minimaxH3SEEDHUNTERLatent_v25.json`)? Where are the files written, how big are they, and do they survive ComfyUI restarts?

crowmaki added the

wayfinder:research

label

2026-10-06 03:55:55 +00:00

crowmaki referenced this issue

2026-10-06 03:56:04 +00:00

Shipped workflow roster and slot contracts #17

crowmaki referenced this issue

2026-10-06 03:56:08 +00:00

Wayfinder: AI Studio end-to-end slice spec #1

crowmaki self-assigned this

2026-10-06 03:56:44 +00:00

crowmaki referenced this issue from a commit

2026-10-06 04:00:46 +00:00

research: reusing H3 AV latents across ComfyUI jobs (#14)

crowmaki commented

2026-10-06 04:00:59 +00:00

Author

Owner

Resolution

Yes, the latents can be reused across jobs and across restarts, but you cannot feed the joint AV latent straight into core SaveLatent. H3 latents are a NestedTensor((video, audio)). SaveLatent calls samples["samples"].contiguous() and writes it with safetensors, which fails on a NestedTensor.

Working recipe (core nodes only):

Draft job: take the selected first-pass denoised_output (the value at ImpactSwitch 122), send it through LTXVSeparateAVLatent, and save the result with 2× SaveLatent. This gives one video.latent and one audio.latent.
Promotion job:
1. Load both files with 2× LoadLatent, using the name plus the [output] suffix. VALIDATE_INPUTS skips the input-dir combo check, so no copy is needed.
2. Send the video to MinimaxH3LatentUpscaler3D.
3. Join the result and the audio with LTXVConcatAVLatent.
4. Run the 2nd Pass.

Files: output/<prefix>_<NNNNN>_.latent. The default prefix is latents/ComfyUI. Each file is safetensors with keys latent_tensor and latent_format_version_0, and its header holds the prompt and workflow JSON (unless --disable-metadata is set). Read the real filename from the job history, because the counter increments. The files are plain files, so they survive restarts. AI Studio should copy them into the Rendition folder and not rely on ComfyUI's output dir.

Size: float32. At the workflow's default 0.2 MP / 8 s draft, it is about 4.6 MB of video and 80 KB of audio. At 768×1344 it is about 14 MB for 5 s and 41 MB for 15 s.

Caveats:

SaveLatent saves only samples. The noise mask is lost. If the draft forced custom audio (SetLatentNoiseMask with SolidMask 0), re-apply that mask to the loaded audio. If you do not, the 0.4-denoise 2nd pass re-noises the audio.
The conditioning is not in the latent. Rebuild MiniMaxH3ReferenceToVideo → MiniMaxH3RefModApply with the same prompt, refs and size.
Alternative: Tr1dae's ComfyUI-MiniMaxH3_LatentUpscaler Save/Load Package. It is not installed. It stores the video, audio, both masks and the conditioning (conditioning.pt, a pickle) in output/h3_lq_stash/.

Not run in ComfyUI. One experiment would settle it:

Run a single job with the upscale pass enabled.
Run the split version: draft + Separate + 2×SaveLatent, then restart ComfyUI, then a LoadLatent-based Promotion graph.
Compare the outputs.

Context pointer

Full findings with source citations (file and line in ComfyUI 0.38.2 and the custom nodes, plus a trace of the workflow graph) are on branch research/h3-latent-reuse, file research/h3-latent-reuse.md.

## Resolution **Yes, the latents can be reused across jobs and across restarts, but you cannot feed the joint AV latent straight into core `SaveLatent`.** H3 latents are a `NestedTensor((video, audio))`. `SaveLatent` calls `samples["samples"].contiguous()` and writes it with safetensors, which fails on a NestedTensor. **Working recipe (core nodes only):** - **Draft job:** take the selected first-pass `denoised_output` (the value at ImpactSwitch 122), send it through `LTXVSeparateAVLatent`, and save the result with 2× `SaveLatent`. This gives one `video.latent` and one `audio.latent`. - **Promotion job:** 1. Load both files with 2× `LoadLatent`, using the name plus the ` [output]` suffix. `VALIDATE_INPUTS` skips the input-dir combo check, so no copy is needed. 2. Send the video to `MinimaxH3LatentUpscaler3D`. 3. Join the result and the audio with `LTXVConcatAVLatent`. 4. Run the 2nd Pass. **Files:** `output/<prefix>_<NNNNN>_.latent`. The default prefix is `latents/ComfyUI`. Each file is safetensors with keys `latent_tensor` and `latent_format_version_0`, and its header holds the `prompt` and workflow JSON (unless `--disable-metadata` is set). Read the real filename from the job history, because the counter increments. The files are plain files, so they survive restarts. AI Studio should copy them into the Rendition folder and not rely on ComfyUI's output dir. **Size:** float32. At the workflow's default 0.2 MP / 8 s draft, it is about 4.6 MB of video and 80 KB of audio. At 768×1344 it is about 14 MB for 5 s and 41 MB for 15 s. **Caveats:** - `SaveLatent` saves only `samples`. **The noise mask is lost.** If the draft forced custom audio (SetLatentNoiseMask with SolidMask 0), re-apply that mask to the loaded audio. If you do not, the 0.4-denoise 2nd pass re-noises the audio. - **The conditioning is not in the latent.** Rebuild `MiniMaxH3ReferenceToVideo` → `MiniMaxH3RefModApply` with the same prompt, refs and size. - **Alternative:** Tr1dae's `ComfyUI-MiniMaxH3_LatentUpscaler` Save/Load Package. It is not installed. It stores the video, audio, both masks and the conditioning (`conditioning.pt`, a pickle) in `output/h3_lq_stash/`. **Not run in ComfyUI.** One experiment would settle it: 1. Run a single job with the upscale pass enabled. 2. Run the split version: draft + Separate + 2×SaveLatent, then restart ComfyUI, then a LoadLatent-based Promotion graph. 3. Compare the outputs. ## Context pointer Full findings with source citations (file and line in ComfyUI 0.38.2 and the custom nodes, plus a trace of the workflow graph) are on branch `research/h3-latent-reuse`, file `research/h3-latent-reuse.md`.

crowmaki closed this issue

2026-10-06 04:00:59 +00:00

crowmaki referenced this issue

2026-10-06 04:01:22 +00:00

Verify latent round trip and refine-pass cost #20

crowmaki referenced this issue

2026-10-07 07:24:15 +00:00

Spec: AI Studio end-to-end slice #24

crowmaki referenced this issue

2026-10-07 07:34:50 +00:00

Promotion and re-promotion #57

crowmaki referenced this issue