Problem
Disclosure first: I work on Sonilo, a licensed video-to-music API, so this proposal involves our product. I'm also a regular HyperFrames user; my own compositions render through hyperframes-cli, and this proposal comes straight out of that loop.
Every bgm route today chooses the music from text before the render exists. Retrieval searches the catalog by mood query. The Lyria/MusicGen fallback generates from an inferred prompt (the keyword tables in inferBgmPrompt()). Both work, but neither one knows where the scene changes land or where the energy picks up. /music-to-video covers the reverse direction, where a track drives the cut. Nothing goes video-first, so on motion-heavy pieces the bed and the picture only line up by luck.
Proposed solution
An optional post-render pass. Send the rendered MP4 to Sonilo's video_to_music endpoint, get back an original track generated from that render (length matched automatically), and mix it at the usual bed volume so narration stays clear; the existing BGM_BED_VOLUME mix already does the right thing. Tracks are licensed and safe for commercial use (terms apply).
Shape is your call. It could be a third bgm.mode next to retrieve and generate, or a post-render step in the pipeline. media-use already treats non-HeyGen generators as opt-in alternatives (Kokoro, mflux, LTX), so this would follow the same pattern: off by default, user-supplied API key, skipped without one, and bgm failure still never blocks a render.
There's also a video_to_sfx endpoint that returns royalty-free sound effects timed to the picture, which could be a later follow-up for compositions where naming SFX per line means guessing.
Alternatives considered
Status quo: I do this loop by hand today (render, generate the track from the MP4, mux with ffmpeg). It works, it just lives outside the pipeline and every agent workflow has to reinvent it. A skill instead of an engine change is also plausible, if you'd rather keep vendors out of media-use.
Additional context
Integration is one API call plus the mix you already have. General recipe: https://github.com/cindyxu1030/sonilo-video-to-music-cookbook. The same post-render step was merged into MoneyPrinterTurbo and NarratoAI this month.
Since HyperFrames is agent-native, one note: there's an MCP server (github.com/sonilo-ai/sonilo-mcp) if that reads as a cleaner surface than an engine provider.
Would this be welcome? Happy to open the PR once the shape sounds right to you.
Problem
Disclosure first: I work on Sonilo, a licensed video-to-music API, so this proposal involves our product. I'm also a regular HyperFrames user; my own compositions render through hyperframes-cli, and this proposal comes straight out of that loop.
Every bgm route today chooses the music from text before the render exists. Retrieval searches the catalog by mood query. The Lyria/MusicGen fallback generates from an inferred prompt (the keyword tables in
inferBgmPrompt()). Both work, but neither one knows where the scene changes land or where the energy picks up./music-to-videocovers the reverse direction, where a track drives the cut. Nothing goes video-first, so on motion-heavy pieces the bed and the picture only line up by luck.Proposed solution
An optional post-render pass. Send the rendered MP4 to Sonilo's
video_to_musicendpoint, get back an original track generated from that render (length matched automatically), and mix it at the usual bed volume so narration stays clear; the existingBGM_BED_VOLUMEmix already does the right thing. Tracks are licensed and safe for commercial use (terms apply).Shape is your call. It could be a third
bgm.modenext toretrieveandgenerate, or a post-render step in the pipeline. media-use already treats non-HeyGen generators as opt-in alternatives (Kokoro, mflux, LTX), so this would follow the same pattern: off by default, user-supplied API key, skipped without one, and bgm failure still never blocks a render.There's also a
video_to_sfxendpoint that returns royalty-free sound effects timed to the picture, which could be a later follow-up for compositions where naming SFX per line means guessing.Alternatives considered
Status quo: I do this loop by hand today (render, generate the track from the MP4, mux with ffmpeg). It works, it just lives outside the pipeline and every agent workflow has to reinvent it. A skill instead of an engine change is also plausible, if you'd rather keep vendors out of media-use.
Additional context
Integration is one API call plus the mix you already have. General recipe: https://github.com/cindyxu1030/sonilo-video-to-music-cookbook. The same post-render step was merged into MoneyPrinterTurbo and NarratoAI this month.
Since HyperFrames is agent-native, one note: there's an MCP server (github.com/sonilo-ai/sonilo-mcp) if that reads as a cleaner surface than an engine provider.
Would this be welcome? Happy to open the PR once the shape sounds right to you.