feat(linux/vaapi): VAAPI encoder improvements

[psyke83]

Description

• Split rate control selection logic out from VBV bitrate enforcement to make it possible for user-overrides.
• Add rate control mode selection. If set to a supported mode, the built-in whitelist is disabled. If set to Automatic or the user selects a mode that is not supported by their driver, the preexisting logic that was used still applies: if whitelisted, try VBR -> CBR -> CQP and force single-frame VBV, otherwise just try CBR -> CQP.
• Add encoder quality profile selection. As VAAPI doesn't mandate specific quality levels, we must probe the maximum level during initialization. If VAConfigAttribEncQualityRange returns a value, the maximum value will represent the fastest/lowest quality and value 1 sets highest quality. This value is then mapped onto the compression_level AVOption which sets the VAEncMiscParameterTypeQualityLevel via ffmpeg's vaapi_encode.c.
• Add Block Level Bitrate Control (BLBRC) option that can be combined with VBR or CBR rate control on supported encoders.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

See our AI usage policy.

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[Kishi85]

I'm getting the following error during encoder probing (causing a fallback to vulkan) when enabling BLBRC while using fe4d3b2:

Jul 07 17:38:28 desktop sunshine[22467]: [2026-07-07 17:38:28.180]: Error: [h264_vaapi @ 0x55e8cfc00780] Unable to parse "blbrc" option value "59746582" as boolean
Jul 07 17:38:28 desktop sunshine[22467]: [2026-07-07 17:38:28.180]: Error: [h264_vaapi @ 0x55e8cfc00780] Error setting option blbrc to value 59746582.

[psyke83]

@Kishi85

The latest rebase should include the necessary fix. I can't test BLBRC on my AMD card, so let me know if it causes any other issues when enabled.

[Kishi85]

Been running this for a few hours yesterday with all options (VBR, BLBRC, STRICT) flipped on and quality set to high.

Subjectively I'm impressed with the quality I'm getting especially on red-heavy scenes with movement it's a lot less washed out compared to Vulkan encoding. Unsure where that comes from. Also host processing latency is just about .2 ms higher than current Vulkan default options which is negliable for me (1.9ms->2.1ms average).

[psyke83]

I added a full drop-down menu for rate control and included all possible modes.

The default RC behaviour should now be exactly the same as before:

• Automatic (default) -> standard probe that includes RC / VBV whitelist enforcement.
• User selects unsupported RC mode -> falls back to same whitelist-compliant probe.
• User selects supported RC mode -> whitelist is overridden for VBV, allowing users to disable strict VBV.

Unfortunately this added a lot of code, most of which is borrowed from the Windows AMF encoder logic and UI section, but I suppose I do prefer the UI category layout and named options now.

[ReenigneArcher]

Can you fix the doxygen errors?

Generating code/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:415: error: Documentation for enum member 'config::vaapi::quality_e::_auto' is missing.
--
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:416: error: Documentation for enum member 'config::vaapi::quality_e::speed' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:417: error: Documentation for enum member 'config::vaapi::quality_e::balanced' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:419: error: Documentation for enum member 'config::vaapi::quality_e::quality' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:425: error: Documentation for enum member 'config::vaapi::rc_e::_auto' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:426: error: Documentation for enum member 'config::vaapi::rc_e::avbr' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:427: error: Documentation for enum member 'config::vaapi::rc_e::cbr' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:428: error: Documentation for enum member 'config::vaapi::rc_e::cqp' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:429: error: Documentation for enum member 'config::vaapi::rc_e::icq' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:430: error: Documentation for enum member 'config::vaapi::rc_e::qvbr' is missing.
/home/docs/checkouts/readthedocs.org/user_builds/sunshinestream/checkouts/5388/src/config.cpp:432: error: Documentation for enum member 'config::vaapi::rc_e::vbr' is missing.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud