fix(transcoder): codec-aware decoder so an HEVC source stops crash-looping the subprocess

[ntt0601zcoder]

Problem (audit A-3, HIGH)

The transcoder built an H.264 decoder at startup and never changed it. A raw-TS source (UDP / HLS-pull / SRT / file) whose video is HEVC — or an established stream whose upstream mux switches H.264→HEVC mid-session — fed H.265 NAL units into the H.264 decoder, which returns AVERROR_INVALIDDATA on every packet. The supervisor escalated that to a terminal error → respawn → same first frame → respawn again: a permanent per-stream crash loop (restart_count climbing, no transcoded output). On NVENC hosts the cuvid parser instead produced silent black / audio-only. Either mode was permanent; only a manual restart "recovered" (into the same loop).

Fix

Make decoder selection codec-aware. Both decode paths (ProcessPacket AV-path and decodeAndEncodeESFrames raw-TS path) now call ensureVideoDecoder(codec) before handing the frame to the decoder. When the active decoder's family (decoderCodecFamily) differs from the incoming codec it:

1. flushes the old decoder through the surviving encoder so the cutover is monotonic in encoder PTS space,
2. rebuilds via newDecoderWithFallback using videoDecoderNameForCodec — esCodecH265 → hevc / hevc_cuvid, preserving the GPU (cuvid) vs CPU lane of the current decoder; cuvid degrades to CPU if NVDEC can't open it,
3. rebinds the GPU scale graphs to the new decoder's CUDA pool (MarkSourceChanged, same as switchInputGPU),
4. resets the keyframe gate so decode resumes on the new codec's first IRAP.

The encoder is never rebuilt — it operates on decoded YUV/CUDA frames and is codec-agnostic — so the output stream stays continuous. This handles HEVC-from-first-frame and a mid-stream H.264→HEVC switch. The AV-path keyframe gate is now codec-aware too (isVideoKeyframeAnnexB → HEVC IRAP detection); the raw-TS path already tagged f.keyframe per codec at demux time.

If HEVC decode is genuinely unavailable in the linked libavcodec, the rebuild surfaces one descriptive error (decoder "hevc" not available…) instead of the opaque invalid-data respawn loop.

Test

TestDecoderCodecFamily, TestVideoDecoderNameForCodec (incl. GPU/CPU-lane preservation), TestEnsureVideoDecoder_NoRebuildPaths (hot-path no-op decisions), TestIsVideoKeyframeAnnexB (codec-aware gate + dispatch divergence).

go test -race ./internal/transcoder/native/ green, golangci-lint 0 issues, full go build ./... green.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 44.73684% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/transcoder/native/stream_pipeline.go	26.31%	40 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!