crowmaki/ai-studio

Watch

Fork 0

MCP server: drive the running app from an AI harness #157

New issue

Closed

opened 2026-10-10 10:55:52 +00:00 by crowmaki · 0 comments

crowmaki commented

2026-10-10 10:55:52 +00:00

Owner

Problem Statement

A filmmaker who works with an AI harness (Claude Code or a similar agent) wants the harness to do the repetitive production work: take a screenplay, build the references, plan coverage, render and review Takes, and hand an edit to DaVinci Resolve. The harness can already drive Blender through a Blender MCP and Resolve through a Resolve MCP. AI Studio, which sits in the middle of that pipeline, has no programmatic surface at all. Every Character, Reference set, Setup, Shot, Keyframe, Take and export has to be clicked through by hand in the app's UI, so the agent stops at AI Studio's front door and the filmmaker does the middle of the film by hand.

Solution

While AI Studio is running, it serves an MCP server on the local machine. The harness connects to it the same way it connects to the Blender and Resolve MCPs, and drives the open Project through tools named in the app's own vocabulary (Scene, Setup, Shot, Take, Tile, Lock snapshot, Keyframe, Edit package…).

Through those tools the harness can carry a film from script to Edit package:

Create or open a Project and import the script.
Fill in Characters, Looks, Wardrobe items, Locations and Props, and their identity and description text.
Generate and Edit Reference versions for Tiles, import images made elsewhere (for example a Blender render), and lock Reference sets.
Cast Scenes, add Setups and Shots, set action text, prompts and Slot lists.
Generate, Edit or import Keyframes, including frames taken from other Takes.
Import or Bounce Performances and adjust Line marks.
Queue Takes, wait for the Render queue, look at the results, circle, pin, NG and note Takes, re-roll stale ones and promote Takes to finals.
Set the Cut and export the Edit package, whose timeline the harness then opens in Resolve through the Resolve MCP.

Everything the harness does goes through the same domain commands the UI uses. So the open windows update live as the agent works, every edit lands in the undo log with its usual label, and the filmmaker can step in at any time, in the UI or by asking the agent.

The MCP server is off by default. The filmmaker turns it on in App settings, which shows the address, the access token and a ready-to-paste client configuration.

Hand-drawn Boards are not part of the MCP surface.

User Stories

Connecting

As a filmmaker, I want to turn the MCP server on and off in App settings, so that nothing listens on my machine unless I want it to.
As a filmmaker, I want the MCP server to listen only on the loopback interface, so that no other machine on my network can drive my Project.
As a filmmaker, I want the MCP server to require an access token, so that a random local process or a web page in my browser can't drive the app.
As a filmmaker, I want to see and copy the server's URL and token in App settings, so that I can add it to my harness quickly.
As a filmmaker, I want a "copy client configuration" button that gives me a ready-to-paste snippet (including a claude mcp add command line), so that I don't hand-write JSON.
As a filmmaker, I want to regenerate the token, so that I can revoke a harness I no longer trust.
As a filmmaker, I want to choose the port, so that I can avoid a clash with another local server.
As a filmmaker, I want a clear message in App settings if the port is already in use, so that I know why my harness can't connect.
As a filmmaker, I want the MCP setting to be app-wide and outside any Project's undo log, so that it behaves like the other App settings.
As a filmmaker, I want the status bar to show that the MCP server is on and whether a client is connected, so that I always know when an agent can act on my Project.
As an AI harness, I want server instructions on connect that describe the app's pipeline order and glossary, so that I use the tools in a sensible order and with the right terms.

Project

As an AI harness, I want to read the open Project (title, path, aspect ratio, spoken language, image model, Generation profile), so that I know what I'm working on.
As an AI harness, I want every Project tool to fail with a clear "no Project open" error when none is open, so that I know to open or create one.
As an AI harness, I want to create a new Project at a path, so that I can start a film from nothing.
As an AI harness, I want to open an existing Project by path, so that I can continue a film.
As a filmmaker, I want an agent's request to open or create a Project to go through the app's usual open path, which asks me first if a job is rendering, so that an agent can never pull a rendering Project out from under me.
As an AI harness, I want the open or create tool to report whether the window opened the Project or the filmmaker declined, so that I don't act on the wrong Project.
As an AI harness, I want to set the Project's title, author, aspect ratio, spoken language, image model and Generation profile, so that the film is set up before I render.
As an AI harness, I want to read the ComfyUI preflight (Ready / Not ready with reasons, Offline), so that I can tell the filmmaker what is missing before queuing work.
As an AI harness, I want to read the Generation profile choices ComfyUI offers, so that I pick model files and LoRAs that exist.

Script

As an AI harness, I want to import an OSF script from a path and get the import summary, so that the Project has its Scenes, Lines and Characters.
As an AI harness, I want to read the script as Scenes with their Lines (ids, kinds, text, characters), so that I can plan coverage from it.
As an AI harness, I want to read only production Scenes, so that I skip omitted ones.
As an AI harness, I want to set a Scene's Lighting and Ambience, so that its Location references and sound bed match.
As an AI harness, I want to set the language and text of a SUBTITLE Line, so that foreign-language dialogue is prompted correctly.
As an AI harness, I want to set a Line's duration and the reading rates, so that the Animatic's timing is right before any Performance exists.

Entities

As an AI harness, I want to list Characters, Locations, Location folders and Props, so that I know which Entities exist.
As an AI harness, I want to read one Character with its Looks, effective identity, Wardrobe items and voice set, so that I can fill in what is missing.
As an AI harness, I want to create, rename and delete Characters, Looks, Wardrobe items, Locations and Props, so that I can complete what script import missed.
As an AI harness, I want to create a Prop or non-speaking Character from a Line's text, so that it is linked to where the script mentions it.
As an AI harness, I want to set a Character's identity fields and a Look's overrides, so that prompts describe each subject correctly.
As an AI harness, I want to set a Location's description and Ambience and a Prop's description, so that prompts and references have the right text.
As an AI harness, I want to merge duplicate Characters or Locations, so that near-duplicates from parsing are cleaned up.
As an AI harness, I want to organize Locations into Location folders, so that the filmmaker sees a tidy Entities sidebar.
As an AI harness, I want to manage Lighting, its phrases and the slug table, so that imported time slugs map to the right Lighting.

References

As an AI harness, I want to read a Reference set with its Tiles, each Tile's versions and its Lock snapshots, so that I know what is generated, current and locked.
As an AI harness, I want to list a Location's Reference sets per Lighting and their notes, so that I can scout each Location for each Lighting.
As an AI harness, I want to add and remove Tiles, so that a Location's angles match the Setups that will see them.
As an AI harness, I want to generate a Reference version for a Tile from a prompt and get its job id, so that I can create references with the Project's image model.
As an AI harness, I want to Edit a Tile from an instruction and an ordered list of reference images (any image the Project holds), with an optional seed, so that I can fix a reference without starting over.
As an AI harness, I want to import an image file from disk as a new Reference version of a Tile, so that I can use images made elsewhere, such as a Blender render.
As an AI harness, I want to import a Source sheet from a path, so that one sheet can feed several Tiles.
As an AI harness, I want to import a voice clip into a Character's voice set, so that Takes have a Voice reference.
As an AI harness, I want to lock a Reference set and roll it back to an earlier Lock snapshot, so that only approved references feed production.
As an AI harness, I want to preview the Atlas a group of Looks would make, so that I can check it before rendering.

Viewing images

As a multimodal AI harness, I want to see any image the Project holds (Reference version, Source sheet, Keyframe, Atlas, Panel) as MCP image content, so that I can judge it before I lock it or use it.
As an AI harness, I want images returned downscaled to a sensible maximum size by default, so that looking at one doesn't fill my context.
As an AI harness, I want to ask for an image written at full resolution to a path I give, so that I can pass it to another tool, such as Blender or Resolve.

Coverage

As an AI harness, I want to read and change a Scene's cast and each cast member's Look and Wardrobe item, so that the right appearance is used in that Scene.
As an AI harness, I want to read a Scene's Setups and Shots with their Lines, lengths, subjects and labels, so that I see the current coverage plan.
As an AI harness, I want to add, delete and reorder Setups and set each one's framing note, location angle and subjects, so that I can plan coverage like on a set.
As an AI harness, I want to add Shots over runs of Lines, set their Lines, length and aspect ratio, split them and delete them, so that every Shot fits H3's 15-second limit.
As an AI harness, I want to read and set a Shot's action text and summary, so that each Shot's beat is described in the filmmaker's terms.
As an AI harness, I want to read a Shot's composed prompt, set a Prompt override, and run and finish a reconcile, so that I can hand-tune prompts like the filmmaker does.
As an AI harness, I want to read and set a Shot's Slot lists, so that I can choose which references H3 gets.
As an AI harness, I want to read and set a Setup's Atlas groups and size, so that the Looks are packed sensibly.

Keyframes

As an AI harness, I want to read a Shot's Keyframes and the jobs that are making new ones, so that I know what pins the Shot's look.
As an AI harness, I want to generate a Keyframe at an anchor from a prompt, so that I can pin a Shot's start or end.
As an AI harness, I want to Edit a Keyframe from an instruction and references, so that I can correct it while keeping identity.
As an AI harness, I want to import a Keyframe from an image file, so that I can use a still rendered in Blender.
As an AI harness, I want to take the first or last frame of a Take as another Shot's Keyframe, so that I can continue action across a cut.
As an AI harness, I want to remove a Keyframe, so that a Shot can go back to being driven only by references.

Performance

As an AI harness, I want to read a Scene's Performance with its Line marks, so that I know how dialogue timing drives its Shots.
As an AI harness, I want to import a Performance from an audio file, so that I can use dialogue recorded or synthesized elsewhere.
As an AI harness, I want to set a Performance's Line marks and rerun auto-placement, so that each Shot gets the right audio slice.
As an AI harness, I want to Bounce a Scene's Cut into a new Performance, so that later Takes match the circled ones' dialogue timing.
As an AI harness, I want to switch a Scene's Performance off and on, so that I can let H3 generate dialogue for a Scene.

Rendering

As an AI harness, I want to queue a draft Take or a Single-pass final for a Shot, with an optional seed and Generation profile, so that I can render coverage.
As an AI harness, I want the queue tool to tell me why a Shot can't be queued (broken slot tag, Shot over covered and uncovered Lines…), so that I can fix it.
As an AI harness, I want to read the Render queue and the controls (paused, finals held), so that I know what will run when.
As an AI harness, I want to pause and resume the queue, hold finals, hold a job, move a job, run jobs next, cancel a job and dismiss a failed job, so that I can manage the queue the way the filmmaker does.
As an AI harness, I want a tool that waits until the given jobs finish (or a timeout I choose runs out) and returns their states, so that I don't poll in a tight loop.
As an AI harness, I want a wait that returns early when a job fails, ComfyUI goes offline or the queue is paused, so that I can react instead of waiting for the full timeout.
As an AI harness, I want job progress reported as MCP progress notifications while I wait, if my client supports them, so that the filmmaker sees the render moving.
As an AI harness, I want to retry a Rendition's failed post-steps, so that a missing proxy can be rebuilt.

Review

As an AI harness, I want to read a Shot's Takes and a Scene's Takes with render status, kind, seed, circled, pinned, NG, seen and stale flags, so that I can review coverage.
As a multimodal AI harness, I want to see a Take as a small set of frames sampled across its length, returned as image content, so that I can judge motion and content without a video decoder.
As an AI harness, I want each Rendition's file path (master and proxy), so that another tool, such as Resolve, can open the video directly.
As an AI harness, I want to circle, uncircle, pin, unpin and NG Takes, so that I can mark what the edit should use.
As an AI harness, I want to add and remove Take notes, with an optional time, so that my review is recorded where the filmmaker will see it.
As a filmmaker, I want an agent's review not to mark Takes seen, so that "unseen" still means I haven't watched them myself.
As an AI harness, I want to read a Take's Stale reasons and accept them, so that I only re-roll Takes whose change matters.
As an AI harness, I want to re-roll a Take with a new seed (with current or original inputs), re-roll with the same seed, rebuild a Take, and run the stale re-roll plan for a Scene or a Reference set, so that coverage stays current.
As an AI harness, I want to retry a failed Take and delete failed Takes, so that the queue doesn't fill with dead jobs.
As an AI harness, I want to promote one or many drafts to finals, so that the circled Takes get full quality.
As an AI harness, I want to set Take Line marks, so that the rough cut follows the Take's real timing.

Cut and export

As an AI harness, I want to read a Scene's Animatic (each stretch of Lines, its Shot and the Take it plays), so that I can check the rough cut.
As an AI harness, I want to assign a Line to a Shot in the Cut or reset it, so that I can shape the rough cut.
As an AI harness, I want to export the Edit package (all non-NG Takes or circled Takes only) and get the folder and the timeline file paths back, so that I can import the cut into Resolve through the Resolve MCP.
As an AI harness, I want export progress as MCP progress notifications, so that a long export doesn't look stuck.

Safety and collaboration

As a filmmaker, I want every agent edit in the Project's undo log with its normal label, so that I can undo an agent's change from the Edit menu.
As an AI harness, I want undo, redo and the undo state as tools, so that I can take back my own mistake.
As a filmmaker, I want the open windows to update live as the agent works, so that I can watch and step in.
As a filmmaker, I want edits I make in the UI while the agent works to be visible to the agent on its next read, so that we don't overwrite each other blindly.
As a filmmaker, I want actions that can't be undone and only free space (Purge, Compact, emptying _Removed/, deleting Source sheets) to be left out of the MCP surface, so that an agent can never destroy media.
As an AI harness, I want every domain error returned as a tool error with the app's message, so that I can correct my input and try again.
As an AI harness, I want invalid ids and unknown enum values (aspect ratio, image model, anchor) rejected with the list of valid values where there is one, so that I can fix the call in one step.

Implementation Decisions

Second adapter, not new domain. The MCP server is a new module in the Tauri adapter layer, next to the taurpc Api. Each tool calls the existing ApiImpl command (or the same studio-core functions those commands call). No domain rule is written in the MCP layer, in line with ADR-0003. Where a tool needs something the RPC surface lacks, the command is added to ApiImpl first and the MCP tool calls it, so the UI can use it too.
Shared session. The MCP server holds a clone of the same ApiImpl the windows use, so it shares the session, the Render worker, the undo log and the change hooks. Agent edits emit the same ChangedDto events, so windows refetch as they do for their own edits.
Transport. Streamable HTTP on 127.0.0.1 only, using the official Rust MCP SDK (rmcp), running on the app's existing Tokio runtime. stdio is not offered, because the app is a GUI process that is already running when the harness connects.
Access. Each install has a random bearer token, kept with the App settings. Requests without it are refused. Requests with an Origin header that isn't a loopback origin are refused, against DNS rebinding.
App settings. New app-wide settings: MCP on/off (default off), port (default fixed, editable), and token (regenerate). The App settings dialog shows the URL, the token, a copy button for a client configuration snippet, and an error if the port can't be bound. Changing a setting restarts the listener. These settings are not in any Project's undo log.
Status bar. A small MCP indicator appears when the server is on, showing whether a client session is active.
Tool design. A curated, task-shaped set of tools in glossary vocabulary, in snake_case (for example queue_take, edit_tile, lock_reference_set, export_edit_package). It is not a mechanical mirror of all RPCs. Ids are the same strings the RPC uses. Results are compact JSON (as structured content plus a text copy) built from the existing DTOs. Read tools that return large trees (the script, a Scene's Takes) take a Scene or Shot filter.
Project open and create. These tools go through the same requested-project path as an OS double-click, so the window opens the Project and asks first if a job is rendering. The tool waits (bounded) until the Project is current and reports opened or declined. With no window to ask, it opens directly.
Images out. An image-viewing tool takes a typed image reference (Reference version, Source sheet, Keyframe, Atlas, Panel, Take frame). It returns MCP image content downscaled to a default maximum long edge, with an optional size. With an optional save_to path, it writes the full-resolution image there and returns the path instead. Take frames are sampled across the Rendition with the app's existing ffmpeg support. The number of frames can be chosen up to a cap.
Files in. Tools that bring media into the Project (script, Source sheet, Tile image, Keyframe, voice clip, Performance) take an absolute local path. The harness and the app run on the same machine. There is no base64 upload.
Importing an image as a Tile version. This uses the existing import path (a Source sheet plus a whole-image crop committed to the Tile), so its provenance is an ordinary imported Reference version.
Waiting. wait_for_jobs(job_ids, timeout_seconds) blocks until every job has finished or failed, or the timeout runs out, with a capped maximum. It returns early on a failure, on ComfyUI going offline and on a paused queue. When the client sends a progress token, running-job progress is sent as MCP progress notifications, from the same progress feed the Queue panel follows. Export progress is reported the same way.
Seen is the filmmaker's. No MCP tool marks Takes seen, and viewing Take frames through MCP does not mark them.
Excluded commands. Purge, purge plan, suggestions, Compact, empty _Removed/, delete Source sheets, window commands, the crop editor session, Board save, describe or edit, Notation editing, and App-default settings are not exposed. Authored deletes (Character, Setup, Shot…) are exposed because they are undoable and follow ADR-0004.
Errors. CommandError maps to an MCP tool error (isError) with the app's message and its kind (not found, invalid input, no Project open, conflict…). Errors are never protocol errors, so the agent can recover. Invalid enum inputs list the valid values.
Server instructions. The server sends instructions on initialize: a short description of the pipeline order (script → Entities → references and locks → cast and coverage → Keyframes and Performance → Takes → review → Cut → Edit package), the key glossary terms, and the rule that only Lock snapshots feed production.
Undo labels. Agent edits use the same undo labels as UI edits, so "Undo Circle 12A-3" reads the same whoever circled it.
Blender and Resolve. These are coupled only through files. Blender renders come in through the Keyframe and Tile image imports. Rendition paths and the Edit package's timeline files go out to Resolve.

Testing Decisions

One seam: the MCP protocol boundary. Tests start the MCP server in-process on an ephemeral loopback port, over an ApiImpl with a temp config dir, pointed at fake-comfy. A real MCP client (the rmcp client) connects with the token and calls tools. Tests assert only on tool results, tool errors, returned image content and files, and the ChangedDto events captured through Hooks. They never assert on internals.
What to cover:
- connection: a missing or wrong token is refused, a non-loopback Origin is refused, and the server instructions are present
- "no Project open" errors, then create and open
- script import and read-back
- Entity create, update and merge, with change events
- Tile generate → wait_for_jobs → view image → Edit with references → lock
- Keyframe generate, import and take-from-frame
- queue_take refusal reasons, and queue → wait → Take listed with Rendition paths and frames
- circle, NG and notes, with no Take marked seen
- stale reasons → re-roll
- promote
- Edit package export returns real paths
- undo through MCP reverts an agent edit and emits a change
- excluded tools are absent from tools/list
- wait_for_jobs returns early on a failed job and on a paused queue, and times out cleanly
Prior art: src-tauri/tests/tile_generation.rs (drives ApiImpl with fake-comfy, an install_everything helper and a Hooks sink) and src-tauri/tests/smoke.rs. The MCP tests follow the same setup with an MCP client on top. Domain rules stay covered by the existing studio-core tests and are not re-tested through MCP.
App settings UI: the toggle, the URL and token display, and the port error are checked in the real app with the run-ai-studio skill, as for other settings dialogs.

Out of Scope

Hand-drawn Boards: Panel sketching, saving Boards, Notation, and Board describe and edit through MCP.
Blender previz playblasts as Shot video references. That is #139. Its "attach playblast" and "Shot scene spec" tools are a follow-up once #139 lands.
MCP prompt templates (Extract props, Propose coverage…) from the early plan. Tools only for now.
MCP resources and subscriptions. Change notifications to the agent are not sent, and the agent re-reads.
Remote access (non-loopback), multi-user access, and OAuth.
A stdio transport or a stdio shim.
Storage-reclaiming and other irreversible actions (Purge, Compact, emptying _Removed/).
Driving the UI itself (navigating Panes, opening windows).
Any LLM inside the app. The app still holds no API keys or agent configuration.

Further Notes

The surface is large. Implementation can be sliced vertically: (1) server, settings, auth, Project and script reads, (2) Entities and references with image viewing, (3) coverage and Keyframes, (4) rendering, waiting and review, (5) Performance, Cut and Edit package. Each slice is testable through the same seam.
rmcp must stay on the same Tokio runtime the app enters at startup, because taurpc and the Render worker already share it.
Pin rmcp to an exact version, like the existing specta pin, because the SDK is still moving.

## Problem Statement A filmmaker who works with an AI harness (Claude Code or a similar agent) wants the harness to do the repetitive production work: take a screenplay, build the references, plan coverage, render and review Takes, and hand an edit to DaVinci Resolve. The harness can already drive Blender through a Blender MCP and Resolve through a Resolve MCP. AI Studio, which sits in the middle of that pipeline, has no programmatic surface at all. Every Character, Reference set, Setup, Shot, Keyframe, Take and export has to be clicked through by hand in the app's UI, so the agent stops at AI Studio's front door and the filmmaker does the middle of the film by hand. ## Solution While AI Studio is running, it serves an MCP server on the local machine. The harness connects to it the same way it connects to the Blender and Resolve MCPs, and drives the open Project through tools named in the app's own vocabulary (Scene, Setup, Shot, Take, Tile, Lock snapshot, Keyframe, Edit package…). Through those tools the harness can carry a film from script to Edit package: 1. Create or open a Project and import the script. 2. Fill in Characters, Looks, Wardrobe items, Locations and Props, and their identity and description text. 3. Generate and Edit Reference versions for Tiles, import images made elsewhere (for example a Blender render), and lock Reference sets. 4. Cast Scenes, add Setups and Shots, set action text, prompts and Slot lists. 5. Generate, Edit or import Keyframes, including frames taken from other Takes. 6. Import or Bounce Performances and adjust Line marks. 7. Queue Takes, wait for the Render queue, look at the results, circle, pin, NG and note Takes, re-roll stale ones and promote Takes to finals. 8. Set the Cut and export the Edit package, whose timeline the harness then opens in Resolve through the Resolve MCP. Everything the harness does goes through the same domain commands the UI uses. So the open windows update live as the agent works, every edit lands in the undo log with its usual label, and the filmmaker can step in at any time, in the UI or by asking the agent. The MCP server is off by default. The filmmaker turns it on in App settings, which shows the address, the access token and a ready-to-paste client configuration. Hand-drawn Boards are not part of the MCP surface. ## User Stories ### Connecting 1. As a filmmaker, I want to turn the MCP server on and off in App settings, so that nothing listens on my machine unless I want it to. 2. As a filmmaker, I want the MCP server to listen only on the loopback interface, so that no other machine on my network can drive my Project. 3. As a filmmaker, I want the MCP server to require an access token, so that a random local process or a web page in my browser can't drive the app. 4. As a filmmaker, I want to see and copy the server's URL and token in App settings, so that I can add it to my harness quickly. 5. As a filmmaker, I want a "copy client configuration" button that gives me a ready-to-paste snippet (including a `claude mcp add` command line), so that I don't hand-write JSON. 6. As a filmmaker, I want to regenerate the token, so that I can revoke a harness I no longer trust. 7. As a filmmaker, I want to choose the port, so that I can avoid a clash with another local server. 8. As a filmmaker, I want a clear message in App settings if the port is already in use, so that I know why my harness can't connect. 9. As a filmmaker, I want the MCP setting to be app-wide and outside any Project's undo log, so that it behaves like the other App settings. 10. As a filmmaker, I want the status bar to show that the MCP server is on and whether a client is connected, so that I always know when an agent can act on my Project. 11. As an AI harness, I want server instructions on connect that describe the app's pipeline order and glossary, so that I use the tools in a sensible order and with the right terms. ### Project 12. As an AI harness, I want to read the open Project (title, path, aspect ratio, spoken language, image model, Generation profile), so that I know what I'm working on. 13. As an AI harness, I want every Project tool to fail with a clear "no Project open" error when none is open, so that I know to open or create one. 14. As an AI harness, I want to create a new Project at a path, so that I can start a film from nothing. 15. As an AI harness, I want to open an existing Project by path, so that I can continue a film. 16. As a filmmaker, I want an agent's request to open or create a Project to go through the app's usual open path, which asks me first if a job is rendering, so that an agent can never pull a rendering Project out from under me. 17. As an AI harness, I want the open or create tool to report whether the window opened the Project or the filmmaker declined, so that I don't act on the wrong Project. 18. As an AI harness, I want to set the Project's title, author, aspect ratio, spoken language, image model and Generation profile, so that the film is set up before I render. 19. As an AI harness, I want to read the ComfyUI preflight (Ready / Not ready with reasons, Offline), so that I can tell the filmmaker what is missing before queuing work. 20. As an AI harness, I want to read the Generation profile choices ComfyUI offers, so that I pick model files and LoRAs that exist. ### Script 21. As an AI harness, I want to import an OSF script from a path and get the import summary, so that the Project has its Scenes, Lines and Characters. 22. As an AI harness, I want to read the script as Scenes with their Lines (ids, kinds, text, characters), so that I can plan coverage from it. 23. As an AI harness, I want to read only production Scenes, so that I skip omitted ones. 24. As an AI harness, I want to set a Scene's Lighting and Ambience, so that its Location references and sound bed match. 25. As an AI harness, I want to set the language and text of a SUBTITLE Line, so that foreign-language dialogue is prompted correctly. 26. As an AI harness, I want to set a Line's duration and the reading rates, so that the Animatic's timing is right before any Performance exists. ### Entities 27. As an AI harness, I want to list Characters, Locations, Location folders and Props, so that I know which Entities exist. 28. As an AI harness, I want to read one Character with its Looks, effective identity, Wardrobe items and voice set, so that I can fill in what is missing. 29. As an AI harness, I want to create, rename and delete Characters, Looks, Wardrobe items, Locations and Props, so that I can complete what script import missed. 30. As an AI harness, I want to create a Prop or non-speaking Character from a Line's text, so that it is linked to where the script mentions it. 31. As an AI harness, I want to set a Character's identity fields and a Look's overrides, so that prompts describe each subject correctly. 32. As an AI harness, I want to set a Location's description and Ambience and a Prop's description, so that prompts and references have the right text. 33. As an AI harness, I want to merge duplicate Characters or Locations, so that near-duplicates from parsing are cleaned up. 34. As an AI harness, I want to organize Locations into Location folders, so that the filmmaker sees a tidy Entities sidebar. 35. As an AI harness, I want to manage Lighting, its phrases and the slug table, so that imported time slugs map to the right Lighting. ### References 36. As an AI harness, I want to read a Reference set with its Tiles, each Tile's versions and its Lock snapshots, so that I know what is generated, current and locked. 37. As an AI harness, I want to list a Location's Reference sets per Lighting and their notes, so that I can scout each Location for each Lighting. 38. As an AI harness, I want to add and remove Tiles, so that a Location's angles match the Setups that will see them. 39. As an AI harness, I want to generate a Reference version for a Tile from a prompt and get its job id, so that I can create references with the Project's image model. 40. As an AI harness, I want to Edit a Tile from an instruction and an ordered list of reference images (any image the Project holds), with an optional seed, so that I can fix a reference without starting over. 41. As an AI harness, I want to import an image file from disk as a new Reference version of a Tile, so that I can use images made elsewhere, such as a Blender render. 42. As an AI harness, I want to import a Source sheet from a path, so that one sheet can feed several Tiles. 43. As an AI harness, I want to import a voice clip into a Character's voice set, so that Takes have a Voice reference. 44. As an AI harness, I want to lock a Reference set and roll it back to an earlier Lock snapshot, so that only approved references feed production. 45. As an AI harness, I want to preview the Atlas a group of Looks would make, so that I can check it before rendering. ### Viewing images 46. As a multimodal AI harness, I want to see any image the Project holds (Reference version, Source sheet, Keyframe, Atlas, Panel) as MCP image content, so that I can judge it before I lock it or use it. 47. As an AI harness, I want images returned downscaled to a sensible maximum size by default, so that looking at one doesn't fill my context. 48. As an AI harness, I want to ask for an image written at full resolution to a path I give, so that I can pass it to another tool, such as Blender or Resolve. ### Coverage 49. As an AI harness, I want to read and change a Scene's cast and each cast member's Look and Wardrobe item, so that the right appearance is used in that Scene. 50. As an AI harness, I want to read a Scene's Setups and Shots with their Lines, lengths, subjects and labels, so that I see the current coverage plan. 51. As an AI harness, I want to add, delete and reorder Setups and set each one's framing note, location angle and subjects, so that I can plan coverage like on a set. 52. As an AI harness, I want to add Shots over runs of Lines, set their Lines, length and aspect ratio, split them and delete them, so that every Shot fits H3's 15-second limit. 53. As an AI harness, I want to read and set a Shot's action text and summary, so that each Shot's beat is described in the filmmaker's terms. 54. As an AI harness, I want to read a Shot's composed prompt, set a Prompt override, and run and finish a reconcile, so that I can hand-tune prompts like the filmmaker does. 55. As an AI harness, I want to read and set a Shot's Slot lists, so that I can choose which references H3 gets. 56. As an AI harness, I want to read and set a Setup's Atlas groups and size, so that the Looks are packed sensibly. ### Keyframes 57. As an AI harness, I want to read a Shot's Keyframes and the jobs that are making new ones, so that I know what pins the Shot's look. 58. As an AI harness, I want to generate a Keyframe at an anchor from a prompt, so that I can pin a Shot's start or end. 59. As an AI harness, I want to Edit a Keyframe from an instruction and references, so that I can correct it while keeping identity. 60. As an AI harness, I want to import a Keyframe from an image file, so that I can use a still rendered in Blender. 61. As an AI harness, I want to take the first or last frame of a Take as another Shot's Keyframe, so that I can continue action across a cut. 62. As an AI harness, I want to remove a Keyframe, so that a Shot can go back to being driven only by references. ### Performance 63. As an AI harness, I want to read a Scene's Performance with its Line marks, so that I know how dialogue timing drives its Shots. 64. As an AI harness, I want to import a Performance from an audio file, so that I can use dialogue recorded or synthesized elsewhere. 65. As an AI harness, I want to set a Performance's Line marks and rerun auto-placement, so that each Shot gets the right audio slice. 66. As an AI harness, I want to Bounce a Scene's Cut into a new Performance, so that later Takes match the circled ones' dialogue timing. 67. As an AI harness, I want to switch a Scene's Performance off and on, so that I can let H3 generate dialogue for a Scene. ### Rendering 68. As an AI harness, I want to queue a draft Take or a Single-pass final for a Shot, with an optional seed and Generation profile, so that I can render coverage. 69. As an AI harness, I want the queue tool to tell me why a Shot can't be queued (broken slot tag, Shot over covered and uncovered Lines…), so that I can fix it. 70. As an AI harness, I want to read the Render queue and the controls (paused, finals held), so that I know what will run when. 71. As an AI harness, I want to pause and resume the queue, hold finals, hold a job, move a job, run jobs next, cancel a job and dismiss a failed job, so that I can manage the queue the way the filmmaker does. 72. As an AI harness, I want a tool that waits until the given jobs finish (or a timeout I choose runs out) and returns their states, so that I don't poll in a tight loop. 73. As an AI harness, I want a wait that returns early when a job fails, ComfyUI goes offline or the queue is paused, so that I can react instead of waiting for the full timeout. 74. As an AI harness, I want job progress reported as MCP progress notifications while I wait, if my client supports them, so that the filmmaker sees the render moving. 75. As an AI harness, I want to retry a Rendition's failed post-steps, so that a missing proxy can be rebuilt. ### Review 76. As an AI harness, I want to read a Shot's Takes and a Scene's Takes with render status, kind, seed, circled, pinned, NG, seen and stale flags, so that I can review coverage. 77. As a multimodal AI harness, I want to see a Take as a small set of frames sampled across its length, returned as image content, so that I can judge motion and content without a video decoder. 78. As an AI harness, I want each Rendition's file path (master and proxy), so that another tool, such as Resolve, can open the video directly. 79. As an AI harness, I want to circle, uncircle, pin, unpin and NG Takes, so that I can mark what the edit should use. 80. As an AI harness, I want to add and remove Take notes, with an optional time, so that my review is recorded where the filmmaker will see it. 81. As a filmmaker, I want an agent's review not to mark Takes seen, so that "unseen" still means I haven't watched them myself. 82. As an AI harness, I want to read a Take's Stale reasons and accept them, so that I only re-roll Takes whose change matters. 83. As an AI harness, I want to re-roll a Take with a new seed (with current or original inputs), re-roll with the same seed, rebuild a Take, and run the stale re-roll plan for a Scene or a Reference set, so that coverage stays current. 84. As an AI harness, I want to retry a failed Take and delete failed Takes, so that the queue doesn't fill with dead jobs. 85. As an AI harness, I want to promote one or many drafts to finals, so that the circled Takes get full quality. 86. As an AI harness, I want to set Take Line marks, so that the rough cut follows the Take's real timing. ### Cut and export 87. As an AI harness, I want to read a Scene's Animatic (each stretch of Lines, its Shot and the Take it plays), so that I can check the rough cut. 88. As an AI harness, I want to assign a Line to a Shot in the Cut or reset it, so that I can shape the rough cut. 89. As an AI harness, I want to export the Edit package (all non-NG Takes or circled Takes only) and get the folder and the timeline file paths back, so that I can import the cut into Resolve through the Resolve MCP. 90. As an AI harness, I want export progress as MCP progress notifications, so that a long export doesn't look stuck. ### Safety and collaboration 91. As a filmmaker, I want every agent edit in the Project's undo log with its normal label, so that I can undo an agent's change from the Edit menu. 92. As an AI harness, I want undo, redo and the undo state as tools, so that I can take back my own mistake. 93. As a filmmaker, I want the open windows to update live as the agent works, so that I can watch and step in. 94. As a filmmaker, I want edits I make in the UI while the agent works to be visible to the agent on its next read, so that we don't overwrite each other blindly. 95. As a filmmaker, I want actions that can't be undone and only free space (Purge, Compact, emptying `_Removed/`, deleting Source sheets) to be left out of the MCP surface, so that an agent can never destroy media. 96. As an AI harness, I want every domain error returned as a tool error with the app's message, so that I can correct my input and try again. 97. As an AI harness, I want invalid ids and unknown enum values (aspect ratio, image model, anchor) rejected with the list of valid values where there is one, so that I can fix the call in one step. ## Implementation Decisions - **Second adapter, not new domain.** The MCP server is a new module in the Tauri adapter layer, next to the taurpc `Api`. Each tool calls the existing `ApiImpl` command (or the same `studio-core` functions those commands call). No domain rule is written in the MCP layer, in line with ADR-0003. Where a tool needs something the RPC surface lacks, the command is added to `ApiImpl` first and the MCP tool calls it, so the UI can use it too. - **Shared session.** The MCP server holds a clone of the same `ApiImpl` the windows use, so it shares the session, the Render worker, the undo log and the change hooks. Agent edits emit the same `ChangedDto` events, so windows refetch as they do for their own edits. - **Transport.** Streamable HTTP on `127.0.0.1` only, using the official Rust MCP SDK (`rmcp`), running on the app's existing Tokio runtime. stdio is not offered, because the app is a GUI process that is already running when the harness connects. - **Access.** Each install has a random bearer token, kept with the App settings. Requests without it are refused. Requests with an `Origin` header that isn't a loopback origin are refused, against DNS rebinding. - **App settings.** New app-wide settings: MCP on/off (default off), port (default fixed, editable), and token (regenerate). The App settings dialog shows the URL, the token, a copy button for a client configuration snippet, and an error if the port can't be bound. Changing a setting restarts the listener. These settings are not in any Project's undo log. - **Status bar.** A small MCP indicator appears when the server is on, showing whether a client session is active. - **Tool design.** A curated, task-shaped set of tools in glossary vocabulary, in snake_case (for example `queue_take`, `edit_tile`, `lock_reference_set`, `export_edit_package`). It is not a mechanical mirror of all RPCs. Ids are the same strings the RPC uses. Results are compact JSON (as structured content plus a text copy) built from the existing DTOs. Read tools that return large trees (the script, a Scene's Takes) take a Scene or Shot filter. - **Project open and create.** These tools go through the same requested-project path as an OS double-click, so the window opens the Project and asks first if a job is rendering. The tool waits (bounded) until the Project is current and reports `opened` or `declined`. With no window to ask, it opens directly. - **Images out.** An image-viewing tool takes a typed image reference (Reference version, Source sheet, Keyframe, Atlas, Panel, Take frame). It returns MCP image content downscaled to a default maximum long edge, with an optional size. With an optional `save_to` path, it writes the full-resolution image there and returns the path instead. Take frames are sampled across the Rendition with the app's existing ffmpeg support. The number of frames can be chosen up to a cap. - **Files in.** Tools that bring media into the Project (script, Source sheet, Tile image, Keyframe, voice clip, Performance) take an absolute local path. The harness and the app run on the same machine. There is no base64 upload. - **Importing an image as a Tile version.** This uses the existing import path (a Source sheet plus a whole-image crop committed to the Tile), so its provenance is an ordinary imported Reference version. - **Waiting.** `wait_for_jobs(job_ids, timeout_seconds)` blocks until every job has finished or failed, or the timeout runs out, with a capped maximum. It returns early on a failure, on ComfyUI going offline and on a paused queue. When the client sends a progress token, running-job progress is sent as MCP progress notifications, from the same progress feed the Queue panel follows. Export progress is reported the same way. - **Seen is the filmmaker's.** No MCP tool marks Takes seen, and viewing Take frames through MCP does not mark them. - **Excluded commands.** Purge, purge plan, suggestions, Compact, empty `_Removed/`, delete Source sheets, window commands, the crop editor session, Board save, describe or edit, Notation editing, and App-default settings are not exposed. Authored deletes (Character, Setup, Shot…) are exposed because they are undoable and follow ADR-0004. - **Errors.** `CommandError` maps to an MCP tool error (`isError`) with the app's message and its kind (not found, invalid input, no Project open, conflict…). Errors are never protocol errors, so the agent can recover. Invalid enum inputs list the valid values. - **Server instructions.** The server sends instructions on initialize: a short description of the pipeline order (script → Entities → references and locks → cast and coverage → Keyframes and Performance → Takes → review → Cut → Edit package), the key glossary terms, and the rule that only Lock snapshots feed production. - **Undo labels.** Agent edits use the same undo labels as UI edits, so "Undo Circle 12A-3" reads the same whoever circled it. - **Blender and Resolve.** These are coupled only through files. Blender renders come in through the Keyframe and Tile image imports. Rendition paths and the Edit package's timeline files go out to Resolve. ## Testing Decisions - **One seam: the MCP protocol boundary.** Tests start the MCP server in-process on an ephemeral loopback port, over an `ApiImpl` with a temp config dir, pointed at `fake-comfy`. A real MCP client (the `rmcp` client) connects with the token and calls tools. Tests assert only on tool results, tool errors, returned image content and files, and the `ChangedDto` events captured through `Hooks`. They never assert on internals. - **What to cover:** - connection: a missing or wrong token is refused, a non-loopback `Origin` is refused, and the server instructions are present - "no Project open" errors, then create and open - script import and read-back - Entity create, update and merge, with change events - Tile generate → `wait_for_jobs` → view image → Edit with references → lock - Keyframe generate, import and take-from-frame - `queue_take` refusal reasons, and queue → wait → Take listed with Rendition paths and frames - circle, NG and notes, with no Take marked seen - stale reasons → re-roll - promote - Edit package export returns real paths - undo through MCP reverts an agent edit and emits a change - excluded tools are absent from `tools/list` - `wait_for_jobs` returns early on a failed job and on a paused queue, and times out cleanly - **Prior art:** `src-tauri/tests/tile_generation.rs` (drives `ApiImpl` with `fake-comfy`, an `install_everything` helper and a `Hooks` sink) and `src-tauri/tests/smoke.rs`. The MCP tests follow the same setup with an MCP client on top. Domain rules stay covered by the existing `studio-core` tests and are not re-tested through MCP. - **App settings UI:** the toggle, the URL and token display, and the port error are checked in the real app with the run-ai-studio skill, as for other settings dialogs. ## Out of Scope - Hand-drawn Boards: Panel sketching, saving Boards, Notation, and Board describe and edit through MCP. - Blender previz playblasts as Shot video references. That is #139. Its "attach playblast" and "Shot scene spec" tools are a follow-up once #139 lands. - MCP prompt templates (Extract props, Propose coverage…) from the early plan. Tools only for now. - MCP resources and subscriptions. Change notifications to the agent are not sent, and the agent re-reads. - Remote access (non-loopback), multi-user access, and OAuth. - A stdio transport or a stdio shim. - Storage-reclaiming and other irreversible actions (Purge, Compact, emptying `_Removed/`). - Driving the UI itself (navigating Panes, opening windows). - Any LLM inside the app. The app still holds no API keys or agent configuration. ## Further Notes - The surface is large. Implementation can be sliced vertically: (1) server, settings, auth, Project and script reads, (2) Entities and references with image viewing, (3) coverage and Keyframes, (4) rendering, waiting and review, (5) Performance, Cut and Edit package. Each slice is testable through the same seam. - `rmcp` must stay on the same Tokio runtime the app enters at startup, because taurpc and the Render worker already share it. - Pin `rmcp` to an exact version, like the existing `specta` pin, because the SDK is still moving.

crowmaki added the

enhancement

ready-for-agent

labels

Rows
Columns