ComfyUI API surface #13

New issue

Closed

opened 2026-10-06 03:55:54 +00:00 by crowmaki · 1 comment

crowmaki commented

2026-10-06 03:55:54 +00:00

Owner

Part of #1

Question

How does an external app drive a local ComfyUI over its HTTP/websocket API? Cover submitting API-format prompts, uploading images and audio, websocket progress and completion events, fetching outputs, interrupting and cancelling, and enumerating installed nodes and models (/object_info, model folders) for the preflight check and the per-Take model/LoRA pickers. Cite primary sources (ComfyUI repo/docs).

Part of #1 ## Question How does an external app drive a local ComfyUI over its HTTP/websocket API? Cover submitting API-format prompts, uploading images and audio, websocket progress and completion events, fetching outputs, interrupting and cancelling, and enumerating installed nodes and models (`/object_info`, model folders) for the preflight check and the per-Take model/LoRA pickers. Cite primary sources (ComfyUI repo/docs).

crowmaki added the

wayfinder:research

label

2026-10-06 03:55:54 +00:00

crowmaki referenced this issue

2026-10-06 03:56:04 +00:00

Shipped workflow roster and slot contracts #17

crowmaki referenced this issue

2026-10-06 03:56:08 +00:00

Wayfinder: AI Studio end-to-end slice spec #1

crowmaki self-assigned this

2026-10-06 03:56:44 +00:00

crowmaki referenced this issue from a commit

2026-10-06 03:59:38 +00:00

research: ComfyUI API surface (#13)

crowmaki commented

2026-10-06 03:59:52 +00:00

Author

Owner

Resolution

Verified against ComfyUI master @ 7c8fbc6 (v0.39.0): server.py, execution.py, main.py, folder_paths.py, and the official script_examples/websockets_api_example.py. Also checked docs.comfy.org comms_routes and comms_messages.

Key facts the spec depends on:

Make calls from Rust, not the webview. Unless ComfyUI runs with --enable-cors-header, its origin middleware returns 403 when a loopback request has Host != Origin or Sec-Fetch-Site: cross-site. That blocks a Tauri webview fetch. Every route also exists with an /api/ prefix.
Submit. POST /api/prompt {prompt: <API-format graph>, client_id, prompt_id?}.
- The client can supply prompt_id; it must be a canonical UUID.
- Validation is synchronous. A 400 response carries error.type, for example missing_node_type or prompt_outputs_failed_validation, plus per-node node_errors.
- A 200 can still carry non-empty node_errors. In that case only the valid outputs are queued, so treat it as a failure.
Upload. POST /api/upload/image (multipart: image, type=input, subfolder, overwrite) handles images and audio, because there is no content-type check. Use the returned name as the node input; a name clash gets a name (1).ext rename. The default size limit is 100 MB (--max-upload-size).
Progress. Open WS /ws?clientId=X. Per-prompt events go only to the socket whose id matches the prompt's client_id.
- Event order: execution_start → execution_cached → executing{node} / progress{value,max} / progress_state → executed{output} → execution_success | execution_error | execution_interrupted.
- executing{node:null, prompt_id} means finished, and history is already written.
- Binary frames hold previews: a u32 big-endian event type followed by the image.
Outputs.
- GET /api/history/{prompt_id} returns outputs[node_id] as UI dicts. Video comes as images + animated:[true], audio as audio, and custom nodes may use other keys, so walk them all for {filename,subfolder,type}.
- Download each file with GET /api/view?filename&subfolder&type=output.
- History is in-memory only (lost on restart). Persist results on our side right away.
Cancel.
- POST /api/jobs/{id}/cancel atomically interrupts the job if it is running, or dequeues it if pending. It is idempotent and present in 0.39.0; feature-detect it.
- Fallback: POST /api/interrupt {prompt_id} (always pass the id; with no body it interrupts globally), plus POST /api/queue {delete:[id]}.
- The queue is shared with the user's own ComfyUI tab, and only one prompt runs at a time.
Preflight and pickers.
- GET /api/object_info returns the installed class_types, so diff it against the template. Combo inputs come in two encodings: legacy [[opts],{}] and V3 ["COMBO",{options}].
- GET /api/models lists the folder keys. GET /api/models/{checkpoints|loras|diffusion_models|…} returns relative filenames that are exactly the values loader combos accept. The list is rescanned by mtime, so no restart is needed.
- GET /api/view_metadata/{folder}?filename= returns safetensors metadata, and GET /api/system_stats returns the version and VRAM.

Context: full findings with line-level citations are on branch research/comfyui-api, file research/comfyui-api.md.

## Resolution Verified against ComfyUI `master` @ `7c8fbc6` (v0.39.0): `server.py`, `execution.py`, `main.py`, `folder_paths.py`, and the official `script_examples/websockets_api_example.py`. Also checked docs.comfy.org comms_routes and comms_messages. **Key facts the spec depends on:** - **Make calls from Rust, not the webview.** Unless ComfyUI runs with `--enable-cors-header`, its origin middleware returns 403 when a loopback request has `Host` != `Origin` or `Sec-Fetch-Site: cross-site`. That blocks a Tauri webview `fetch`. Every route also exists with an `/api/` prefix. - **Submit.** `POST /api/prompt {prompt: <API-format graph>, client_id, prompt_id?}`. - The client can supply `prompt_id`; it must be a canonical UUID. - Validation is synchronous. A 400 response carries `error.type`, for example `missing_node_type` or `prompt_outputs_failed_validation`, plus per-node `node_errors`. - **A 200 can still carry non-empty `node_errors`.** In that case only the valid outputs are queued, so treat it as a failure. - **Upload.** `POST /api/upload/image` (multipart: `image`, `type=input`, `subfolder`, `overwrite`) handles images **and audio**, because there is no content-type check. Use the returned `name` as the node input; a name clash gets a `name (1).ext` rename. The default size limit is 100 MB (`--max-upload-size`). - **Progress.** Open `WS /ws?clientId=X`. Per-prompt events go only to the socket whose id matches the prompt's `client_id`. - Event order: `execution_start` → `execution_cached` → `executing{node}` / `progress{value,max}` / `progress_state` → `executed{output}` → `execution_success` | `execution_error` | `execution_interrupted`. - `executing{node:null, prompt_id}` means finished, and history is already written. - Binary frames hold previews: a u32 big-endian event type followed by the image. - **Outputs.** - `GET /api/history/{prompt_id}` returns `outputs[node_id]` as UI dicts. Video comes as `images` + `animated:[true]`, audio as `audio`, and custom nodes may use other keys, so walk them all for `{filename,subfolder,type}`. - Download each file with `GET /api/view?filename&subfolder&type=output`. - **History is in-memory only (lost on restart).** Persist results on our side right away. - **Cancel.** - `POST /api/jobs/{id}/cancel` atomically interrupts the job if it is running, or dequeues it if pending. It is idempotent and present in 0.39.0; feature-detect it. - Fallback: `POST /api/interrupt {prompt_id}` (always pass the id; with no body it interrupts globally), plus `POST /api/queue {delete:[id]}`. - The queue is **shared** with the user's own ComfyUI tab, and only one prompt runs at a time. - **Preflight and pickers.** - `GET /api/object_info` returns the installed `class_type`s, so diff it against the template. Combo inputs come in two encodings: legacy `[[opts],{}]` and V3 `["COMBO",{options}]`. - `GET /api/models` lists the folder keys. `GET /api/models/{checkpoints|loras|diffusion_models|…}` returns relative filenames that are exactly the values loader combos accept. The list is rescanned by mtime, so no restart is needed. - `GET /api/view_metadata/{folder}?filename=` returns safetensors metadata, and `GET /api/system_stats` returns the version and VRAM. **Context:** full findings with line-level citations are on branch `research/comfyui-api`, file `research/comfyui-api.md`.

crowmaki closed this issue

2026-10-06 03:59:52 +00:00

crowmaki referenced this issue

2026-10-07 07:24:15 +00:00

Spec: AI Studio end-to-end slice #24