Spec: AI Studio end-to-end slice #24
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
needs-info
needs-triage
question
ready-for-agent
ready-for-human
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
crowmaki/ai-studio#24
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Spec for the end-to-end slice charted in #1. Every decision below comes from a closed child ticket of #1 (linked inline). Terms follow
CONTEXT.md. ADRs 0001–0005 apply.Problem Statement
A filmmaker who works alone with a local video model (MiniMax H3 through ComfyUI on one RTX 5090 / 32 GB) has no tool that connects a screenplay to an editor's timeline. Today they hand-build references, hand-write long six-field H3 prompts, keep track in their head of which reference version fed which clip, and rename and sort hundreds of generated files before an editor can cut them. Nothing records why a clip looks the way it does, nothing flags clips made from references that have since changed, and nothing turns "the takes I like" into something DaVinci Resolve understands. The work is slow and error-prone, and it can't be reproduced.
Solution
AI Studio is a local desktop app (Tauri 2 + SvelteKit + Svelte 5, with a Rust
studio-corethat owns the domain). It takes a screenplay through a pipeline organized like real production:OSF script → Entities → locked References → Atlases → Setups / Shots with Boards and Keyframes → Takes (drafts, Promotion, Single-pass final) → Animatic → Resolve Edit package.
The filmmaker imports an OSF script. The app creates Scenes, Lines, Characters and Locations. The filmmaker builds Reference sets by importing and cropping sheets or by generating Tiles, and locks them as Lock snapshots. They plan Setups and Shots, draw Boards with a structured Notation, and queue Takes. The app composes the H3 prompt, fills the reference slots, builds Atlases, and runs a render queue that it owns against the user's own ComfyUI. Takes come back to Dailies, where the filmmaker screens, circles, notes and promotes them. The Animatic plays the Scene's Cut, and circled Takes replace Boards until it is a rough cut. An optional Performance locks dialogue timing. Export writes an Edit package: binned
12A-3.movclips, FCPXML 1.10 and.otiotimelines, and a Resolve metadata CSV with circled status and notes. Every Take keeps full provenance, so it can be explained, judged stale, and reproduced.User Stories
Project and environment
.studiofile with a sibling media folder, so that I can move, back up and double-click a Project like a document.Script import
urn:drafter:1dialect), so that Scenes, Lines, Characters and Locations are created for me.d:idand Lines to get stable opaque IDs, so that my production data survives for the life of the Project.d:ids to get new IDs with a warning, so that a messy script still imports.12A) used as written, and other Scenes numbered in script order and frozen at import, so that Take names never change.2Aworks) and Scene page ranges derived from them, so that I can still find things by page.Entities
References
Atlases
Setups, Shots and the Board
12B·1,12B·2), so that I never send H3 a length it can't make.Prompt composer
<Subject N>and Props as<Prop N>, built from structured fields and never named, so that names don't bias the model.<Picture 2>") with the views listed per Parity, so that H3 maps subjects to rows correctly.Slot budget
Generation and the render queue
12A-3with numbers never reused, so that names are stable for the editor.01:00:00:00timecode and PCM audio, plus an H.264 proxy for in-app playback, so that the editor gets real masters and I can still play them.Dailies
Staleness
Animatic and Cut
Performance
fully_copyaudio slot, so that every Setup matches the locked timing.Edit package
12A-3.movclips (segmented Shots12B_1-3.mov), so that the Media Pool reads like real coverage..movstarting at01:00:00:00(the ProRes master if final, otherwise the draft remuxed with PCM audio), so that clips behave consistently..otio(with markers), so that I can start from the Animatic in Resolve..srt, and per-Scene Performance audio, so that the Animatic is complete in Resolve.takes.csvfor Resolve's metadata import (matched by file name and start/end TC) that sets Scene, Shot, Take,Tag=1for circled, the framing note, keywords (draft/final/stale), notes as Comments, and Seed/Profile/Pinned columns, so that circled status and notes survive into Resolve.manifest.jsonwith each Take's full recorded inputs and the packaged Rendition's id and hash, so that any clip can be explained and regenerated later._Removed/<date>/), using temp files and renames, so that the editor's links keep working and nothing is deleted._Removed/.Storage
_Removed/, Compact) that only select rows, so that nothing runs without me.Implementation Decisions
Architecture (ADR-0003, #3)
studio-corecrate (no Tauri, no specta). It owns the domain model, SQLite, OSF import, Lock snapshot rules, staleness, Atlas composition, the prompt composer, the slot budget, the render queue and worker, the ComfyUI client, ffmpeg calls and export.src-tauriis a thin adapter..studiofile. The frontend changes state only through named domain commands (taurpc). DTOs are defined in the adapter and mapped from the core's view-shaped read models. Commands return a typed error enum.changed { kind, ids }invalidation events are broadcast to all windows, and views refetch.tauri::ipc::Channelcarries per-job progress and progress for long operations (OSF import, export).ProjectSessionper process holds the connection, the undo log and the worker. It may be shown in several windows.studio:custom scheme (async) servesblob/<id>from SQLite andrendition/<id>with Range/206. URLs carry ids, not paths, and the webview has no file scope.imagecrate, and Tile crops are cut from the original sheet. The UI only draws.Project file (ADR-0001, ADR-0004, ADR-0005, #11)
.studiois a SQLite database. Renditions and latents live in a sibling<name>.media/folder under ID-based names (renditions/<rendition-id>.*,latents/<rendition-id>.{video,audio}.latent) with relative paths.d:idas a unique column.blobtable (BLAKE3, media type, size, pixel dimensions). Unused blobs are swept on open and on demand, with no reference counts.PRAGMA user_versionmigrations embedded in the core, applied in one transaction after copying to<name>.studio.bak-v<N>. A newer file is refused. A fixedapplication_id. WAL while open, then checkpoint andjournal_mode=DELETEon clean close. A<name>.studio.lockfile holds PID and host.projectrow. Lists get their own tables (lighting,slug_lighting,reference_category_order,notation_entry). App-wide defaults are a JSON file in the app config directory, copied into a Project at creation and never read again.deleted_at. Unconsumed rows are hard-deleted. Failed Takes may be hard-deleted, and numbers are never reused. Merged Characters and Locations become tombstones withmerged_into.character,locationandproptables. Each Reference set owner holds a unique FK (look,wardrobe_item,prop,character.voice_set_id, andlocation_lighting(location, lighting, reference_set, lighting_note)).reference_set.kindis one of turnaround, mannequin, location, prop or voice. Tiles have role and ordinal (fixed roles for turnaround and mannequin) and are retired, never deleted, once any snapshot includes them.lock_snapshot(set, number)+snapshot_tileis immutable, and the current snapshot is the highest number.atlas(input_key, blob, built_at)is a cache (ADR-0005). The key hashes the ordered snapshot IDs, the layout version and the size settings. Setups store only their grouping.first_line_id/last_line_id.setup(scene, letter, setup_order).shot(setup, index, next_take_number).scene_cast(scene, character, look, wardrobe_item?)andsetup_subject(setup, character, position).text,mention{kind,id},dialogue{line_id}) plus a derivedshot_mentiontable rewritten on save. A merge rewrites both in one transaction. The prompt override is plain text plus its fork-base composed text.shot_slot_item(shot, position, candidate_key, state, is_new)+shot.slot_line, with logical keys (keyframe:<id>,atlas:<group>,wardrobe:<id>,location_angle:<tile>,board,prop:<id>). No rows means the Shot follows the Project order live.panel(board, position, canvas_w, canvas_h, sketch_blob, annotations JSON)with normalized annotation positions. Boards are edited in place with no history.keyframe(shot, anchor, at_seconds?, image_blob, origin). A take_frame Keyframe records its source Take, Rendition and frame time.performance(scene, number, audio_blob, created_at)is immutable.performance_sourceis an open edge list of source kinds (Takes, carried-forward Performance, import).performance_mark(performance, line, start)plus an end, a coverage flag per Line per version, andscene.performance_off.take_inputhas one row per consumed thing, with a real FK and the slot number it filled. Shot and Setup text fields are recorded by value or hash. The exact job graph and prompt text are stored as blobs, plus an overridden flag.take_link(take, other_take, kind)with kindsrerolled_fromandframe_from(andextendsreserved for Take extension). The Take also records its Performance slice and snapshotted marks, seen, NG, notes with optional timecodes, circled-at, accepted reasons, and the shipped workflow id/version/hash.shot.pinned_takemust be circled and is cleared on uncircle.rendition(take, kind, via, state, path?, purged_at, size, hash, poster_blob, refine_seed?), created when queued. Latents have their own purged marker. Missing is computed when Storage opens.jobhas one row per ComfyUI submission: kind, exactly one target FK (Rendition, Reference version or Panel), state,queue_position, attempt, our ownprompt_id, held flag, error and timestamps. The Hold finals switch is persisted on the Project.OSF import (#2)
basestyle. Reads onlyscene_time_separator. Ignores<lists>,<styles>, revision colours and empty paragraphs. Strips inline formatting but keeps soft breaks.References and Atlases (#4, #5, #10)
Prompt composer (#9)
A pure function from a Shot's current inputs plus the filled slot assignment to H3's six-field text, always in this order and with these names:
subject_definitions,summary,retention_analysis,detailed_description,overall_soundscape,non_diegetic_music(alwaysN/A). Exactly one[Shot 1], with noid=header.<Subject N>is numbered by first appearance in the Shot, and<Prop N>by attachment order.<Environment>is the Location.<Picture N>/<Audio N>are numbered in wired order after the budget is filled. No names or pronouns.Retention entries, taken from #9:
fully_preserved - face, hair, eyes, physique and clothing.partially_preserved - … clothing is not inherited.; mannequinattribute_transfer - the <item> is worn by <Subject N>; the mannequin does not appear.fully_preserved - architecture, layout, fixed objects and lighting; the camera angle may differ.fully_preserved - shape, material, wear and markings.weak_reference- rough storyboard of N panels, staging/composition/camera only, style and figures not reproduced, arrows are directionsfully_preserved - first frame at 00:00.000 …/last frame at <ms>reference - its vocal timbre guides the vocal delivery of <Subject N> …fully_copy - the dialogue of this shot …. Voice references are dropped.newly_generated - no visual reference …detailed_description= the framing note, then the Notation read-back sentences in panel order (never naming the storyboard), then the action text with dialogue chips expanded as<Subject N> says: <d>[Language] <tag> text</d>. O.S./O.C. becomes "says from off-screen:", V.O. becomes "says in voice-over:", and dual dialogue becomes "… speak at the same time." Character-naming parentheticals move out of the quote.overall_soundscape= the Location's Ambience + the Scene's Ambience addition.An override is forked from the preview and stores its fork base. A composer change shows only a quiet marker, and the three-way reconcile runs on demand. Slot tags in an override are validated, and a broken one blocks queuing. A Take records the exact text and an overridden flag.
Slot budget (#10)
Shipped workflows and ComfyUI (ADR-0002, #13, #14, #15, #17, #20)
h3.draft,h3.promotion,h3.final,image.t2iandimage.board(both image workflows on Qwen-Image 2.1, andimage.boardwith its own fixed rough-style prompt). The owner's SEEDHUNTER graph is a technique reference only.node_id.input), output nodes, the node pack name/URL for each non-core class_type, and the folder/URL for each fixed model file. Rust makes at most three structural edits: oneLoadImage/LoadAudioper filled slot wired into theMiniMaxH3ReferenceToVideoAutogrow ports in slot order; aLoraLoaderModelOnlychain at a named model point (the chain's position relative to the shipped LoRAs doesn't matter); and, for Performance-driven Takes only, the forced-audio branch (VAEEncodeAudio→ 0 mask). The filled graph is the job blob.h3.promotion. Old versions are not kept.LTXVSeparateAVLatent→ 2×SaveLatent, plus an H.264/AAC MP4 at 24 fps and a lossless FLAC (coreSaveAudio). Saved latent file names are read from history and copied into the media folder.LoadLatent "<file> [output]"→MinimaxH3LatentUpscaler3D→LTXVConcatAVLatent→ refine pass at final size, with conditioning rebuilt from the draft's recorded prompt, slot assignment and blobs. The draft's audio is always frozen with a 0SetLatentNoiseMask. Each Promotion gets a new random refine seed (editable, and recorded on the final)..movmaster written by the workflow. A CPU post-step then stamps01:00:00:00and PCM audio into it and makes an H.264 proxy at short side 768 (h264_nvencif a startup check finds it, otherwiselibopenh264). Every H3 job writes a lossless FLAC.VAEEncodeAudio→ 0 mask) and as thefully_copyslot.WS /ws?clientId,POST /api/upload/imagefor images and audio (use the returned name),POST /api/promptwith our own UUIDprompt_id, where a 200 with non-emptynode_errorsis a failure. Progress follows the documented event order, andexecuting{node:null}means done. Results are fetched from/api/history/{id}(walk every output key) and/api/view, and persisted immediately because history is in memory only. Cancel usesPOST /api/jobs/{id}/cancel(feature-detected), falling back to interrupt-with-id + queue delete. The ComfyUI queue is shared with the user's own tab./system_stats), every class_type (/object_info, both combo encodings), every fixed model file and combo value, and the job's profile files (/api/models/*). The result is Ready or Not ready with reasons. It never installs anything. Pickers read/api/models/*.Render worker (#18)
prompt_id, collected from history, or requeued at the head of its class as an interrupted attempt. Closing with a running job warns and then leaves it running./free(a setting). Offline is a waiting state with re-probing. Not-ready workflows' jobs wait. A failed draft or Single-pass final becomes a Failed Take. A failed Promotion leaves the draft plus a "Promotion failed" note with Retry. A post-step failure keeps the GPU output with a "proxy missing" marker and retry.Staleness (#12, #23)
take_input, recorded text and hashes, slot assignment, Performance version and slice) against the job the Shot would build now, ignoring seed and Generation profile. Each difference is a Stale reason of kind References, Board/Keyframes, Performance (version, marks/slice), Prompt, Framing (including effective aspect ratio) or Workflow (informational only).Dailies (#7)
prototype/dailiesis the visual reference.Animatic and Performance (#8, #23)
silencedetect), the mark partition rules, the coverage set, the straddle block and the ghost lane are as in User Stories 181–191. Re-run auto-placement is one undoable edit.Edit package (#16, #19, #21)
The folder layout, taken from #19:
FCPXML 1.10 is hand-written with exactly one event holding one project, and no browser clips, bins or markers (otherwise Resolve fails with "No valid sequences"). Timeline: the Project ratio at final size, 24 fps, starting at
01:00:00:00, with drafts scaled to fit..otiocarries timeline markers (transitions, shot directions) and clip markers (timecoded notes).takes.csvheaders:File Name,Clip Directory,Start TC,End TC,Scene,Shot,Take,Tag(1for circled, blank otherwise, never2),Description,Keywords,Comments, plus customSeed,Profile,Pinned. UTF-8. Never use theGood Takecolumn.The sync algorithm is driven by the previous manifest, with temp-file writes and renames. Hard links are used on the same volume, otherwise copies.
Storage (#22)
VACUUMwith the queue paused. In-file deletes are undoable within the session, while Purge and Empty_Removedare not.Testing Decisions
studio-core'sProjectSession. Tests drive the same named domain commands and view-shaped read models that thetaurpcadapter wraps (import OSF, crop and lock, set up Shots, queue, collect, circle, promote, bounce, export, purge…). Each test opens a real.studiofile in a temp directory. Tests never read tables or call private modules directly.ProjectSession, and asserts only on what the API returns or on the artifacts the core writes. Examples: the composed prompt text for a Shot; the filled job graph sent to ComfyUI; an Atlas's bytes or hash (golden, because it must be byte-stable); the read model's Stale reasons; the Edit package tree,takes.csv, FCPXML and.otiocontents. Tests don't assert on table layout or internal structs, so the schema can change freely under them./api/prompt(recording each submitted graph andprompt_id),/api/upload/image,/wsevent sequences,/api/history,/api/view,/api/jobs/{id}/cancel,/api/interrupt,/api/object_info,/api/models/*,/system_statsand/free. It returns canned fixtures (a short MP4, ProRes.mov, FLAC, latent files, PNG). It can be scripted to produce: a 400 validation error, a 200 withnode_errors,execution_error, OOM, being offline, a job still running or finished when a Project reopens, history lost after a restart, and missing nodes or models for preflight.libopenh264,silencedetect).src-tauriadapter gets only a smoke test (it is thin DTO mapping).prototype/crop-sheet,prototype/board-paint,prototype/dailies).research/h3-latent-reuse, localscratch/resolve-test/), not in CI.research/h3-latent-reuse(research/h3-latent-reuse-run/), which show the exact draft and Promotion graphs that the job builder's output should match, and the Resolve test kit's reference FCPXML, OTIO and CSV files, which are golden references for export.Out of Scope
These are beyond the slice and come back only as a new effort (from #1):
take_linkkinds, chained inputs) must not rule it out.[Shot 2]blocks, intercuts).performance_sourcemust stay open to new kinds.non_diegetic_musicis alwaysN/A, and music is added in Resolve.Further Notes
keyword/notereach Resolve's Media Pool is unknown, which is why only the CSV is relied on.research/comfyui-api,research/finishing-pass,research/h3-latent-reuseandresearch/resolve-fcpxml. UI prototypes are onprototype/crop-sheet,prototype/board-paintandprototype/dailies. Treat prototypes as visual references only, not code to keep.scratch/scratch-plan.md, is local and gitignored. Where it conflicts with this spec (for example its LLM prop proposal, Fountain import or RIFE), this spec and the closed tickets win.Spec amendments decided while reviewing #79 (#33) and #80 (#35):
VAEEncodeAudio→ 0 mask), and only Performance-driven Takes get it (#65). The position of the userLoraLoaderModelOnlychain relative to the shipped LoRAs doesn't matter.SaveAudio), not WAV. This covers Story 140, the Draft and Finals bullets, and the fake ComfyUI fixtures. The per-ScenePerformance.wavin the Edit package doesn't change.#51 and #65 have been updated to match. The text of this spec body still needs the same edits.
Follow-up from #47: the app-wide default Slot order is now its own ticket, #99 (App Settings: default Slot order). A Project copies the default order only when it is created.
Audited all 215 user stories against
main(90a8019). 209 are implemented. Every child ticket is closed. These gaps remain, each with its own ticket:board_imagessends only the Sketch, so the clean toggle changes only the prompt text.Smaller notes, not ticketed:
defaults.jsonby hand.This spec stays open until #133 to #137 are closed.