Queue recovery and failures #53

New issue

Closed

opened 2026-10-07 07:34:49 +00:00 by crowmaki · 1 comment

crowmaki commented

2026-10-07 07:34:49 +00:00

Owner

Parent

#24

What to build

A crash, restart or ComfyUI problem never loses or duplicates work. See User Stories 130–134.

Acceptance criteria

The queue always reopens paused with a banner; the in-flight job is reattached by prompt_id, collected from history, or requeued at the head of its class as an interrupted attempt
Closing with a job running warns, then leaves it running in ComfyUI
ComfyUI errors fail the job with no retry; OOM retries once after /free (a setting)
ComfyUI offline is a waiting state with re-probing; jobs for a Not-ready workflow wait
A failed draft or Single-pass final becomes a Failed Take
Tests script the fake for: still running on reopen, finished on reopen, history lost, 400, 200 with node_errors, execution_error, OOM and offline

Blocked by

## Parent #24 ## What to build A crash, restart or ComfyUI problem never loses or duplicates work. See User Stories 130–134. ## Acceptance criteria - [ ] The queue always reopens paused with a banner; the in-flight job is reattached by `prompt_id`, collected from history, or requeued at the head of its class as an interrupted attempt - [ ] Closing with a job running warns, then leaves it running in ComfyUI - [ ] ComfyUI errors fail the job with no retry; OOM retries once after `/free` (a setting) - [ ] ComfyUI offline is a waiting state with re-probing; jobs for a Not-ready workflow wait - [ ] A failed draft or Single-pass final becomes a Failed Take - [ ] Tests script the fake for: still running on reopen, finished on reopen, history lost, 400, 200 with `node_errors`, `execution_error`, OOM and offline ## Blocked by - #52

crowmaki added the

enhancement

ready-for-agent

labels

2026-10-07 07:34:49 +00:00

crowmaki referenced this issue

2026-10-08 05:57:12 +00:00

Generate a Tile, the first render job (#39) #89

crowmaki commented

2026-10-08 05:58:24 +00:00

Author

Owner

Note from #39 (PR #89): the render worker runs preflight before each submission. In #39, a job whose workflow is Not ready fails at once: the job goes to failed with the Not-ready reasons as its error, and nothing is sent to ComfyUI. An offline ComfyUI also fails the job ("ComfyUI is not reachable: …").

#24 says that jobs for a Not-ready workflow should wait, and this ticket's acceptance criteria include "ComfyUI offline is a waiting state with re-probing; jobs for a Not-ready workflow wait". So this ticket should replace the failure with a waiting state. The code is in run_job in crates/studio-core/src/worker.rs (the preflight check before client.events). The tests that pin today's behaviour are a_workflow_that_is_not_ready_fails_its_job_without_submitting_it and an_unreachable_comfyui_fails_the_job in crates/studio-core/tests/render_worker.rs.

Note from #39 (PR #89): the render worker runs preflight before each submission. In #39, a job whose workflow is Not ready **fails at once**: the job goes to `failed` with the Not-ready reasons as its error, and nothing is sent to ComfyUI. An offline ComfyUI also fails the job ("ComfyUI is not reachable: …"). #24 says that jobs for a Not-ready workflow should wait, and this ticket's acceptance criteria include "ComfyUI offline is a waiting state with re-probing; jobs for a Not-ready workflow wait". So this ticket should replace the failure with a waiting state. The code is in `run_job` in `crates/studio-core/src/worker.rs` (the `preflight` check before `client.events`). The tests that pin today's behaviour are `a_workflow_that_is_not_ready_fails_its_job_without_submitting_it` and `an_unreachable_comfyui_fails_the_job` in `crates/studio-core/tests/render_worker.rs`.

crowmaki referenced this issue

2026-10-08 23:12:29 +00:00

Queue a draft Take (#51) #105

crowmaki referenced this issue

2026-10-08 23:46:46 +00:00

Queue scheduling and controls (#52) #106

crowmaki self-assigned this

2026-10-08 23:58:11 +00:00

crowmaki referenced this issue from a commit

2026-10-09 00:30:37 +00:00

Recover the render queue after crashes and ComfyUI trouble (#53)

crowmaki referenced this issue from a commit