Queue recovery and failures (#53) #108

Merged

crowmaki merged 3 commits from issue-53-queue-recovery into main

2026-10-09 00:49:01 +00:00

crowmaki commented

2026-10-09 00:30:58 +00:00

Owner

Closes #53. Part of #24 (User Stories 130–134).

What changes

A crash, a restart or trouble with ComfyUI no longer loses work or runs it twice.

Reopening. A Project reopens with its queue paused when work is left in it. The Queue panel shows a banner with Resume, and the dock opens if it was hidden. The worker finds each job that was running by its prompt_id:
- If ComfyUI still runs it, the worker listens again under the same WebSocket client id.
- If it finished, the worker collects it from history.
- If ComfyUI has forgotten it (history lost in a restart), the worker requeues it at the head of its class as an interrupted attempt, under a new prompt_id.
- A job that was claimed but never submitted goes back as it was.
Closing. Closing the last window warns when a job is rendering (Tauri dialog). So do Close / Open / New Project. After you confirm, the job keeps running in ComfyUI. The worker sees that the Project is gone and stops following the job, but it never cancels it.
Failures.
- ComfyUI errors fail the job with no retry: a 400, a 200 with node_errors, or execution_error.
- Running out of GPU memory is the one exception. The worker calls POST /api/free and submits the job once more. A new app setting controls this ("Retry a job once…" in the ComfyUI section, on by default).
Waiting, not failing. Preflight now runs before the worker takes a job.
- While ComfyUI is offline, no job is taken and the queue shows "ComfyUI offline". The worker probes ComfyUI again every 3 s.
- A job whose workflow is Not ready waits, with the reasons on its row. The other jobs run.
- If ComfyUI drops in the middle of a render, the job stays Running until ComfyUI is back. Then the worker reattaches it, collects it or requeues it in the same way.
Failed Take. A failed draft still becomes a Failed Take.
Schema. V25 adds job.client_id, job.waiting_reason and job.interrupted.

Tests

The fake ComfyUI can now hold a prompt until it is released (Block), restart and forget everything, run out of GPU memory, and answer GET /queue and POST /free. tests/queue_recovery.rs uses it to cover:

a job still running on reopen
a job that finished, or was stopped, while the Project was closed
history lost in a restart
a 400, a 200 with node_errors and an execution_error, each with no retry
running out of GPU memory: one retry, a second failure, and the setting switched off
ComfyUI offline at submission and in the middle of a render

The Not-ready tests in render_worker.rs, takes.rs and tile_generation.rs now check that the job waits, and that other Takes render meanwhile. cargo test --workspace --all-features, bun test src and bun run check all pass.

Decisions and known gaps

"Always reopens paused": the queue pauses only when queued or running jobs are left. Reopening with an empty queue does not pause or show the banner, because there is nothing to protect. Please confirm this reading.
Single-pass final → Failed Take: finals don't exist yet. #54 adds them, and they should go through the same land_rendition failure path.
Edge cases not covered:
- A /prompt call that gets no answer while ComfyUI still validates the prompt could be taken for lost and submitted again.
- A job cancelled while the app cannot reach ComfyUI is not stopped later.
- Closing the Project between an OOM /free and the retry resets the retry budget.
Not unit-tested: the close-warning dialogs.

Screenshots

ComfyUI offline: the jobs wait, and the status bar and the Queue panel both say so.

A workflow that is Not ready: the job waits with the reasons on its row.

The Project reopened with work in the queue: the queue is paused, with a banner and Resume.

The new setting in the ComfyUI section.

Closes #53. Part of #24 (User Stories 130–134). ## What changes A crash, a restart or trouble with ComfyUI no longer loses work or runs it twice. - **Reopening.** A Project reopens with its queue paused when work is left in it. The Queue panel shows a banner with **Resume**, and the dock opens if it was hidden. The worker finds each job that was running by its `prompt_id`: - If ComfyUI still runs it, the worker listens again under the same WebSocket client id. - If it finished, the worker collects it from history. - If ComfyUI has forgotten it (history lost in a restart), the worker requeues it at the head of its class as an **interrupted** attempt, under a new `prompt_id`. - A job that was claimed but never submitted goes back as it was. - **Closing.** Closing the last window warns when a job is rendering (Tauri dialog). So do Close / Open / New Project. After you confirm, the job keeps running in ComfyUI. The worker sees that the Project is gone and stops following the job, but it never cancels it. - **Failures.** - ComfyUI errors fail the job with no retry: a 400, a 200 with `node_errors`, or `execution_error`. - Running out of GPU memory is the one exception. The worker calls `POST /api/free` and submits the job once more. A new app setting controls this ("Retry a job once…" in the ComfyUI section, on by default). - **Waiting, not failing.** Preflight now runs before the worker takes a job. - While ComfyUI is offline, no job is taken and the queue shows "ComfyUI offline". The worker probes ComfyUI again every 3 s. - A job whose workflow is Not ready waits, with the reasons on its row. The other jobs run. - If ComfyUI drops in the middle of a render, the job stays Running until ComfyUI is back. Then the worker reattaches it, collects it or requeues it in the same way. - **Failed Take.** A failed draft still becomes a Failed Take. - **Schema.** V25 adds `job.client_id`, `job.waiting_reason` and `job.interrupted`. ## Tests The fake ComfyUI can now hold a prompt until it is released (`Block`), restart and forget everything, run out of GPU memory, and answer `GET /queue` and `POST /free`. `tests/queue_recovery.rs` uses it to cover: - a job still running on reopen - a job that finished, or was stopped, while the Project was closed - history lost in a restart - a 400, a 200 with `node_errors` and an `execution_error`, each with no retry - running out of GPU memory: one retry, a second failure, and the setting switched off - ComfyUI offline at submission and in the middle of a render The Not-ready tests in `render_worker.rs`, `takes.rs` and `tile_generation.rs` now check that the job waits, and that other Takes render meanwhile. `cargo test --workspace --all-features`, `bun test src` and `bun run check` all pass. ## Decisions and known gaps - **"Always reopens paused"**: the queue pauses only when queued or running jobs are left. Reopening with an empty queue does not pause or show the banner, because there is nothing to protect. Please confirm this reading. - **Single-pass final → Failed Take**: finals don't exist yet. #54 adds them, and they should go through the same `land_rendition` failure path. - Edge cases not covered: - A `/prompt` call that gets no answer while ComfyUI still validates the prompt could be taken for lost and submitted again. - A job cancelled while the app cannot reach ComfyUI is not stopped later. - Closing the Project between an OOM `/free` and the retry resets the retry budget. - Not unit-tested: the close-warning dialogs. ## Screenshots ComfyUI offline: the jobs wait, and the status bar and the Queue panel both say so. ![ComfyUI offline](https://git.crowmaki.me/attachments/ba2f448e-ca1e-4ea7-9a65-5fdfeab021dd) A workflow that is Not ready: the job waits with the reasons on its row. ![Not ready](https://git.crowmaki.me/attachments/8141b0ab-dd1d-439f-980f-7f282d3c54a5) The Project reopened with work in the queue: the queue is paused, with a banner and Resume. ![Reopened paused](https://git.crowmaki.me/attachments/6afc30d4-e69b-407e-a8a1-cfdda80472fa) The new setting in the ComfyUI section. ![ComfyUI settings](https://git.crowmaki.me/attachments/8be1c332-085d-4759-9432-1cada34c4cbb)

1-offline.png

62 KiB

2-not-ready.png

71 KiB

3-reopened.png

63 KiB

4-comfyui-settings.png

103 KiB

crowmaki added 3 commits

2026-10-09 00:30:58 +00:00

Recover the render queue after crashes and ComfyUI trouble (#53 ) db9d043188

The queue reopens paused when work was left in it, and the jobs that were
running are reattached by prompt_id, collected from history, or requeued at
the head of their class as interrupted attempts. Closing a Project leaves its
running job running in ComfyUI. An offline ComfyUI and a Not-ready workflow
are waiting states that the worker probes again; running out of GPU memory
retries once after /free, behind an app setting. The fake ComfyUI can now
block a prompt, restart, run out of memory and list its queue.

Show the queue's recovery and waiting states (#53 ) a496072a1f

The Queue panel shows a banner with Resume when the queue reopened paused,
and one while ComfyUI is offline. A job whose workflow is Not ready says why,
and an interrupted job says ComfyUI lost it. The status bar counts Not-ready
jobs and says when ComfyUI is offline. The ComfyUI section switches the retry
after running out of GPU memory, and closing or replacing a Project while a
job renders asks first.

Address review of queue recovery (#53 ) df1128e63e

Claim only jobs preflight found ready, so a job queued during preflight never
skips it. Ask ComfyUI's queue before its history when finding a job again, so
a prompt finishing between the two is not taken for lost. Put back a job that
was claimed but never submitted rather than calling it interrupted. Fix the
close dialog's text, rename unclear names and describe recovery in CONTEXT.

crowmaki merged commit 4f00d81add into main

2026-10-09 00:49:01 +00:00

crowmaki deleted branch issue-53-queue-recovery

2026-10-09 00:49:01 +00:00

crowmaki referenced this pull request from a commit

2026-10-09 00:49:03 +00:00

Merge pull request 'Queue recovery and failures (#53)' (#108) from issue-53-queue-recovery into main