Skip to content

video: required video capability (0xfa) + 0xBEDE RTP extension so sent video renders#3

Open
AstraOnlineWeb wants to merge 2 commits into
purpshell:mainfrom
AstraOnlineWeb:feat/video-rtp-extension-0xbede
Open

video: required video capability (0xfa) + 0xBEDE RTP extension so sent video renders#3
AstraOnlineWeb wants to merge 2 commits into
purpshell:mainfrom
AstraOnlineWeb:feat/video-rtp-extension-0xbede

Conversation

@AstraOnlineWeb

Copy link
Copy Markdown

While testing the video send path against the official WhatsApp clients (Android/iOS), the peer never rendered my video — even though audio and the relay bridge were fine. Comparing my answer-path stanzas and outbound RTP against a capture of a real video call surfaced two missing pieces. With both fixed, the sent camera shows up on the peer.

1. The answer path advertised audio, not video

On an inbound video call, the preaccept/accept were built as if it were an audio call:

  • the capability blob was the audio one — 01 05 f7 09 e4 bb 13. A real client flips byte 6 to 0xfa on a video call (01 05 f7 09 e4 fa 13);
  • no <video> child in the preaccept (and none in the accept unless explicitly requested).

Without the 0xfa capability + the <video> advertisement the callee doesn't bring video up. This also explains the existing sendAccept note that "capability+both-rates fails" — with the audio capability on a video call, it does.

This PR:

  • adds CapabilityVideoOffer (…e4 fa 13) next to the audio CapabilityOffer;
  • preaccept: advertises <video dec="H264" …> and uses the video capability when the offer is video;
  • accept: advertises <video enc="h264"> when the offer is video (via the existing AcceptParams.Video).

2. Every sent video packet needs the RFC 5285 0xBEDE header extension

The official client carries a one-byte header extension on every H.264 RTP packet (profile 0xBEDE): abs-send-time (id 3) + video-orientation/CVO (id 5) + transport-wide-cc seq (id 6). The peer drops video from a sender that omits it; audio is tolerated without it. videoSender was sending none.

This PR:

  • gives RtpHeader a generic extension (ExtensionProfile + ExtensionData) alongside the existing audio 0xdebe word, and has EncodeRtpHeader emit it;
  • adds BuildVideoRtpExtension(tccSeq, cvo) for the abs-send-time + CVO + transport-cc block;
  • has videoSender attach it per packet with a per-packet transport-cc sequence.

Notes

  • The 0xfa capability byte and the 0xBEDE extension layout are taken from real video-call captures. The exact stanza wiring mirrors a working sender (WaCalls).
  • Tests added for the capability byte, the preaccept <video> node, the extension byte layout and the header encoding; go test ./... is green and gofmt clean.
  • Follow-ups I can send separately: pacing the packets (vs. bursting a whole access unit) noticeably speeds how fast the peer locks onto the stream, and stripping AUD (NAL type 9) from the encoder's output since the official client never sends it.

… path

When answering an inbound video call, the preaccept/accept were built as if it
were audio: the audio capability blob (byte 6 0xbb) and no <video> child. A real
client flips that byte to 0xfa and advertises <video> in both the preaccept and
the accept; without it the callee never brings video up (which is why enabling
the audio capability on a video call was failing).

- add CapabilityVideoOffer (…e4 fa 13) alongside the audio CapabilityOffer
- preaccept: advertise <video dec=H264> + the video capability when the offer is video
- accept: advertise <video enc=h264> when the offer is video
- tests for the capability byte and the preaccept node
Every H.264 packet from the official client carries an RFC 5285 one-byte header
extension (profile 0xBEDE): abs-send-time (id 3), video-orientation/CVO (id 5)
and transport-wide-cc seq (id 6). The peer drops video from a sender that omits
it; audio is tolerated without it.

- RtpHeader gains a generic extension (ExtensionProfile + ExtensionData) next to
  the existing audio 0xdebe word; EncodeRtpHeader emits it
- BuildVideoRtpExtension builds the abs-send-time + CVO + transport-cc block
- videoSender attaches it per packet and carries a per-packet transport-cc seq
- tests for the extension layout and header encoding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant