Skip to content

Project Context API — AI reads the timeline + acts on it (with undo) #10

Description

@taskmasterpeace

Part of the agent-native timeline + character design spec (Pillar 2). Today the timeline + transcript live only in the front-end — the backend and the AI can't see them and can't act. This builds the foundation.

Feature goals

  • Read API: an endpoint returning the full current project as JSON — every track (kind video/audio, muted, enabled/visible, locked) and every clip (id, type, startTime, duration, trackIndex, trimStart/End, speed, muted, volume, assetId, isGenerating, linked clips). The model already exists in frontend/types/project.ts; it just isn't exposed.
  • Action API (bounded): placeClip(assetId, trackIndex, startTime), replaceClip(clipId, assetId), move/retime, setMuted(clip|track, bool), removeClip — each one reversible, each pushing a single undo entry.
  • Bridge: the running editor stays source of truth — it publishes a live read-model to the backend and applies queued actions, reporting success/failure (the AI acts in the app the user is watching).

Quality goals

  • Read model accurate to the second (ideally ms), always current (mute/visibility included), audio vs video distinguished.
  • Every action undoable in one Ctrl-Z, never corrupts A/V links, rejects invalid targets with a clear error (never silently wrong).
  • Read is fast enough to call before each AI decision.

Acceptance

The AI can ask "what's on the timeline?" → exact positions/mute/visibility, then run "put asset X at 1:08 on the next free video track and mute track 2" → the user sees it happen, one undo reverts it.

Dependencies

#9 (login) so the agent can authenticate.

Files

frontend/types/project.ts, frontend/views/VideoEditor.tsx, a new backend route/handler for project read + action queue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-nativeAI can perceive/actdesignHas a design specmvpPart of the core AI-native timeline loop

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions