Skip to content

feat(media): video_understand — watch a video (video→text) (v0.140.0)#239

Merged
vikasprogrammer merged 1 commit into
mainfrom
feat/video-understand
Jul 13, 2026
Merged

feat(media): video_understand — watch a video (video→text) (v0.140.0)#239
vikasprogrammer merged 1 commit into
mainfrom
feat/video-understand

Conversation

@vikasprogrammer

Copy link
Copy Markdown
Owner

What

Agents can now "see" a video — the inverse of everything we've built so far (which all generates). Claude can't watch video natively; the new video_understand MCP tool delegates to an Atlas multimodal LLM and returns a text answer.

video_understand({ video: "<artifact id / path / url>", prompt: "what happens at the end?" })
  • video accepts the same three sources as the other media tools: a Library artifact id (e.g. one video_generate just made), a working-folder file (written or terminal-uploaded), or an http(s) URL. Local files inlined as base64 (no hosting needed).
  • prompt optional (default: a detailed description). Also handles stills via kind:"image".
  • Returns text directly to the agent — no artifact.

How

  • Atlas has ~10 catalog LLMs with video in input_modalities (qwen3.5, glm-5v, kimi-k2…). New backend src/edge/media-understand.ts calls the OpenAI-compatible /v1/chat/completions with a {type:"video_url", video_url:{url}} content part (the shape verified to work; {type:"video"} is silently ignored by the models).
  • terminal.understandVideo reuses the media-ref resolver (generalized to kind: 'image'|'video'), gates video.understand (cost estimate → money-cap), audits video.understood.
  • Route POST /api/agent/video/understand; MCP tool exposed whenever Atlas is configured (VIDEO_UNDERSTAND env, set at launch). Default model qwen/qwen3.5-27b.

Verified

  • Live Atlas, in-process through the built adapter: fed a test clip → "A large white rabbit rolls on the grass and then stands up as a carrot falls in front of him." (accurate). Cross-checked two models (qwen3.5-27b, glm-5v-turbo) both describe it correctly.
  • typecheck, build, test:governance (68/68) pass.

Atlas-only. Token-priced (video eats tokens) — the gate uses an estimate; real usage-cost captured when Atlas returns it. New route + MCP tool → server restart + session relaunch to take effect.

🤖 Generated with Claude Code

New governed MCP tool: delegates to an Atlas multimodal LLM (chat
endpoint, video_url content part) and returns a text answer about a
video — Claude can't see video natively. Source is a Library id,
working-folder file, or URL (inlined as base64). New backend
media-understand.ts, route /api/agent/video/understand, capability
video.understand (money-capped). Verified live.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vikasprogrammer vikasprogrammer merged commit c44e14b into main Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant