OpenGL Filter support

[themaxsmith]

Do you plan on exposing GL so you can use OpenGL to apply filters to the streaming video? Most similar libraries on Andriod (https://github.com/pedroSG94/rtmp-rtsp-stream-client-java) and HashinKit on iOS provide this feature or someway to filter/provide a way to modify the broadcast image. I specifically want this feature to be able to add an overlay image via a shader similar to this https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/4b85bce8475deb97380dad3bbe51aca6d24087ea/encoder/src/main/java/com/pedro/encoder/input/gl/render/filters/object/ImageObjectFilterRender.java

As a side note, I think this library has a lot of potentials and think it is heading in the right direction

[ThibaultBee]

Hi,

I am going to be straight: I don't plan to expose OpenGL filters.
StreamPack is designed to focus on performance and video quality (like setting encoder profile and level, I plan to use bidirectionnal frame). So I had to made some compromises for this like having a rather close implementation to the way Android camera API (aka camera2) has been designed. This compromises make OpenGL filters hard to implement.
The main drawback of this implementation is that camera preview is independant from the video stream, that means that the same filter can't be applied for both easily.

I guess you are looking to have an image overlay. Does it matter if it is not displayed on the camera preview? (if the camera is your source).

Moreover, my TODO list is enoooormouuuusss and my time is really limited!

If you have more questions, I will be happy to talk 👍

Glad to hear that you like this library :)

[themaxsmith]

Yeah, it does not need to be displayed on the camera preview just the SRT TS feed. I have experience with OpenGL and SRT streaming (Mostly on the CPP & iOS side). I can look into it if I have more free time. I might consider spending time on adding it to this library if SRT becomes more mainstream vs RTMP. I think it could be possible via encoderSurface. The problem like always is time and the Android user base / lower revenue per user makes it harder for me to justify the time vs iOS.

[ThibaultBee]

Hi,
It would be great if you could have time for this but I totally understand that you have other priorities :)
Thanks anyway for the idea.

[ThibaultBee]

A first step for video effects has been done in 3.0.0-RC3. A new parameter in the StreamerPipeline have been added: the surfaceProcessorFactory. It allows to use a custom ISurfaceProcessor.
The next steps are:

• add a media3 effect surface processor
• make the surfaceProcessorFactory parameter available widely.