Render worker scheduling

crowmaki commented

2026-10-06 03:55:56 +00:00

Owner

Part of #1
Blocked by: #17, #11

Question

How does the app-owned queue choose the next job? Cover priority classes (Promotions ahead of new coverage), model affinity to avoid reloads, pausing and holding H3 jobs for overnight batches, crash recovery, and cancelling or reordering.

Part of #1 Blocked by: #17, #11 ## Question How does the app-owned queue choose the next job? Cover priority classes (Promotions ahead of new coverage), model affinity to avoid reloads, pausing and holding H3 jobs for overnight batches, crash recovery, and cancelling or reordering.

crowmaki added the

wayfinder:grilling

label

2026-10-06 03:55:56 +00:00

crowmaki referenced this issue

2026-10-06 03:56:08 +00:00

Wayfinder: AI Studio end-to-end slice spec #1

crowmaki commented

2026-10-06 05:08:56 +00:00

Author

Owner

Input from Rust core vs TypeScript frontend split: the queue needs an explicit cancel command for queued and in-flight jobs. Undo never reaches queued or rendered work. The worker belongs to the Project session (one Project per process), so closing the Project stops the queue. Decide what happens on reopen (resume, ask, or drop). Per-job progress goes to the UI through tauri::ipc::Channel.

Input from [Rust core vs TypeScript frontend split](https://git.crowmaki.me/crowmaki/ai-studio/issues/3): the queue needs an explicit **cancel** command for queued and in-flight jobs. Undo never reaches queued or rendered work. The worker belongs to the Project session (one Project per process), so closing the Project stops the queue. Decide what happens on reopen (resume, ask, or drop). Per-job progress goes to the UI through `tauri::ipc::Channel`.

crowmaki commented

2026-10-07 05:01:06 +00:00

Author

Owner

From "Staleness rules" (#12): staleness compares a Take's recorded job with the job the Shot would build now. This ticket must decide when the worker builds a job: when it runs, or when it is queued. If it builds when the job runs, a queued Take always picks up the latest inputs and cannot arrive stale. Batch re-roll also skips any Shot that already has a queued or running Take with current inputs, so the queue must be able to answer whether that Take is up to date.

From "Staleness rules" (#12): staleness compares a Take's recorded job with the job the Shot would build *now*. This ticket must decide when the worker builds a job: when it runs, or when it is queued. If it builds when the job runs, a queued Take always picks up the latest inputs and cannot arrive stale. Batch re-roll also skips any Shot that already has a queued or running Take with current inputs, so the queue must be able to answer whether that Take is up to date.

crowmaki commented

2026-10-07 06:12:26 +00:00

Author

Owner

From "Shipped workflow roster and slot contracts" (#17), constraints for scheduling:

Job kinds are h3.draft, h3.promotion, h3.final, image.t2i and image.board. The two image workflows share Qwen-Image 2.1, and the H3 workflows share the H3 model set, so grouping by family avoids model swaps.
Preflight marks each workflow Ready or Not ready. Queued jobs for a Not-ready workflow wait (the worker skips them) and run once a re-check passes. The worker re-runs preflight for the job's workflow and profile files just before each submission.
Re-promotion replaces a Take's final in place, and only once it succeeds. Decide how a re-promotion is queued alongside or behind an existing Promotion for the same Take, and whether a pending one can be superseded.
Measured costs: Promotion at 1984×1120 takes ~90 s for 5 s and ~283 s for 15 s. A single-pass final is a full first pass at final size, so expect it to be slower. Finals also need an ffmpeg proxy step after the ComfyUI job.

From "Shipped workflow roster and slot contracts" (#17), constraints for scheduling: - Job kinds are `h3.draft`, `h3.promotion`, `h3.final`, `image.t2i` and `image.board`. The two image workflows share Qwen-Image 2.1, and the H3 workflows share the H3 model set, so grouping by family avoids model swaps. - Preflight marks each workflow Ready or Not ready. Queued jobs for a Not-ready workflow **wait** (the worker skips them) and run once a re-check passes. The worker re-runs preflight for the job's workflow and profile files just before each submission. - **Re-promotion** replaces a Take's final in place, and only once it succeeds. Decide how a re-promotion is queued alongside or behind an existing Promotion for the same Take, and whether a pending one can be superseded. - Measured costs: Promotion at 1984×1120 takes ~90 s for 5 s and ~283 s for 15 s. A single-pass final is a full first pass at final size, so expect it to be slower. Finals also need an ffmpeg proxy step after the ComfyUI job.

crowmaki self-assigned this

2026-10-07 06:13:49 +00:00

crowmaki commented

2026-10-07 06:24:30 +00:00

Author

Owner

Resolution

When jobs are built

A job is built at queue time. A queued Take already has its number, seed and recorded inputs, so staleness is computed for it like any other Take. Batch re-roll reads it directly to check whether a queued Take is current.
Rebuild with current inputs on a waiting Take keeps its number and seed and only replaces the recorded inputs.

Order

Classes: image jobs (image.t2i, image.board) → drafts (h3.draft) → finals (h3.promotion, h3.final). FIFO within a class.
Model affinity applies within a class only. The worker prefers the oldest job whose checkpoint and quantization match what it last submitted (LoRAs are ignored). It can pass over a job at most ~3 times or for a few minutes. An image job still jumps ahead of H3 drafts and pays the model swap.
Drag to reorder within a class; a drag can't cross classes. Run next (multi-select, keeping the selection's order) puts jobs at the front of the whole queue, and affinity never passes them over.
A running job is never pre-empted.

Holding

Global Pause / Resume stops picking new jobs, and the running job finishes.
Per-job Hold keeps the job in place but skips it.
Hold finals is a queue-wide, sticky switch that holds every Promotion and Single-pass final, including ones queued later. It persists in the Project. There is no scheduled start.

Cancel and re-promotion

Cancel works on queued jobs and on the running one (via ComfyUI interrupt, discarding partial output).
Cancelling a Take that has no Rendition deletes it and burns its number (a per-Shot counter; numbers are never reused).
Cancelling a Promotion or re-promotion leaves the Take with the Renditions it already had.
At most one pending or running final job per Take. Promote and Re-promote are disabled while one exists, and point at it in the queue.

Reopen and crash

The queue persists in .studio and always opens paused on reopen, with a banner ("N jobs waiting · Resume"; after a crash it names the job that was running).
Closing with a job running warns first. If the user confirms, the job is left running in ComfyUI (unless they cancel it).
On reopen, the job that was in flight is reattached by prompt_id if ComfyUI still has it queued or running (even while paused), collected if it finished and is still in history, or requeued at the head of its class (same Take, same seed, an interrupted attempt rather than a failure) if ComfyUI has no record of it.

Failures

ComfyUI errors (node_errors, execution_error) fail the job with no auto-retry. A draft or Single-pass final becomes a Failed Take (Retry/Delete in Dailies). A failed Promotion leaves the Take with its draft plus a "Promotion failed" note with Retry. The worker continues.
OOM: by default, retry once after /free. This is a setting.
ComfyUI unreachable: an offline waiting state, not a failure. Jobs stay queued, the worker re-probes, and on reconnect the in-flight job goes through the reopen paths above. Preflight Not-ready workflows still wait, as decided in #17.
If a post-step fails (ffmpeg proxy, saving), the GPU output is kept. The final Rendition exists with a "proxy missing" marker and a retry action.

Pipelining

The worker owns only the ComfyUI submission lane. Preparation for the next job (Atlas builds, uploads) overlaps the current render. Post-steps run in a separate CPU lane capped at 1–2 at a time, and a Take shows Finishing until they're done. The proxy codec is left to "Edit package layout".

UI

A dockable Queue panel, opened from a status-bar indicator (running job and %, waiting count, Paused / Finals held / ComfyUI offline). It groups jobs by class, with progress, drag, Hold, Run next, Cancel, Rebuild with current inputs, Pause, and the Hold finals switch.
Failed jobs are pinned to the top of the panel until dismissed or until their Take is opened in Dailies.
Status badges on Take tiles in Dailies and Shot listings: Queued / Held / Rendering % / Finishing / Failed.
No history view. Finished jobs leave the panel, and their results live on their Takes.

## Resolution **When jobs are built** - A job is built at **queue time**. A queued Take already has its number, seed and recorded inputs, so staleness is computed for it like any other Take. Batch re-roll reads it directly to check whether a queued Take is current. - **Rebuild with current inputs** on a waiting Take keeps its number and seed and only replaces the recorded inputs. **Order** - Classes: image jobs (`image.t2i`, `image.board`) → drafts (`h3.draft`) → finals (`h3.promotion`, `h3.final`). FIFO within a class. - Model affinity applies **within a class only**. The worker prefers the oldest job whose checkpoint and quantization match what it last submitted (LoRAs are ignored). It can pass over a job at most ~3 times or for a few minutes. An image job still jumps ahead of H3 drafts and pays the model swap. - Drag to reorder within a class; a drag can't cross classes. **Run next** (multi-select, keeping the selection's order) puts jobs at the front of the whole queue, and affinity never passes them over. - A running job is never pre-empted. **Holding** - Global **Pause / Resume** stops picking new jobs, and the running job finishes. - Per-job **Hold** keeps the job in place but skips it. - **Hold finals** is a queue-wide, sticky switch that holds every Promotion and Single-pass final, including ones queued later. It persists in the Project. There is no scheduled start. **Cancel and re-promotion** - Cancel works on queued jobs and on the running one (via ComfyUI interrupt, discarding partial output). - Cancelling a Take that has no Rendition **deletes it and burns its number** (a per-Shot counter; numbers are never reused). - Cancelling a Promotion or re-promotion leaves the Take with the Renditions it already had. - **At most one pending or running final job per Take.** Promote and Re-promote are disabled while one exists, and point at it in the queue. **Reopen and crash** - The queue persists in `.studio` and **always opens paused** on reopen, with a banner ("N jobs waiting · Resume"; after a crash it names the job that was running). - Closing with a job running warns first. If the user confirms, the job is **left running in ComfyUI** (unless they cancel it). - On reopen, the job that was in flight is reattached by `prompt_id` if ComfyUI still has it queued or running (even while paused), collected if it finished and is still in history, or requeued at the head of its class (same Take, same seed, an interrupted attempt rather than a failure) if ComfyUI has no record of it. **Failures** - ComfyUI errors (`node_errors`, `execution_error`) fail the job with **no auto-retry**. A draft or Single-pass final becomes a Failed Take (Retry/Delete in Dailies). A failed Promotion leaves the Take with its draft plus a "Promotion failed" note with Retry. The worker continues. - **OOM**: by default, retry once after `/free`. This is a setting. - ComfyUI unreachable: an **offline** waiting state, not a failure. Jobs stay queued, the worker re-probes, and on reconnect the in-flight job goes through the reopen paths above. Preflight Not-ready workflows still wait, as decided in #17. - If a post-step fails (ffmpeg proxy, saving), the GPU output is kept. The final Rendition exists with a "proxy missing" marker and a retry action. **Pipelining** - The worker owns only the ComfyUI submission lane. Preparation for the next job (Atlas builds, uploads) overlaps the current render. Post-steps run in a separate CPU lane capped at 1–2 at a time, and a Take shows **Finishing** until they're done. The proxy codec is left to "Edit package layout". **UI** - A dockable **Queue panel**, opened from a status-bar indicator (running job and %, waiting count, Paused / Finals held / ComfyUI offline). It groups jobs by class, with progress, drag, Hold, Run next, Cancel, Rebuild with current inputs, Pause, and the Hold finals switch. - **Failed jobs are pinned to the top** of the panel until dismissed or until their Take is opened in Dailies. - Status badges on Take tiles in Dailies and Shot listings: Queued / Held / Rendering % / Finishing / Failed. - No history view. Finished jobs leave the panel, and their results live on their Takes.

crowmaki closed this issue

2026-10-07 06:24:30 +00:00

crowmaki referenced this issue

2026-10-07 06:24:45 +00:00

Edit package layout #19

crowmaki referenced this issue

2026-10-07 07:24:15 +00:00

Spec: AI Studio end-to-end slice #24

crowmaki referenced this issue

2026-10-07 07:34:49 +00:00

Queue scheduling and controls #52

crowmaki referenced this issue