Is there an example of streaming inference with simultaneous audio playback for monologue mode?

[geekgogo]

First of all, thank you for open-sourcing such an excellent project. I have a question that I would like to ask:

When performing monologue streaming inference, the audio saved directly to a file sounds great. However, during real-time playback, there is consistently a strong electrical noise, and I haven’t been able to achieve smooth audio playback while performing inference simultaneously. This issue has troubled me for several days.

May I ask if there is any existing example or recommended approach for achieving streaming inference with smooth real-time audio playback?

Thank you very much!

[ukemamaster]

@geekgogo I think it is related to the fact that the model generated 80ms audio chunk in more than 80ms. So the real time factor is not suitable for real time playback, and some cuts are noticed in the audio. At least this is the case with my NVIDIA A30 gpus.
Could you share your hardware and your generation time details?

[geekgogo]

@ukemamaster Yes, exactly, I have also noticed this on the NVIDIA A100 now, the RTF is greater than 1s.

[ukemamaster]

RFT greater than 1 means you still cant get real time playback even on A100 ?

[geekgogo]

@ukemamaster Yes, the official statement indicates a first packet delay of 140ms and an RTF of 0.8~0.9, which were tested on the L20 GPU.

[ukemamaster]

Yeah.
Have you tried to run inference in VLLM, or SGLang etc?

[geekgogo]

@ukemamaster Not yet, I don't know how to use vLLM and SGLang for acceleration. If you know, could you tell me?

[ukemamaster]

i tried, but it seems not supported.

[wangyuepeng1]

I also encountered this issue. Saving to the local device sounds great, but when playing in real time, there is always a frame loss problem. After reducing to bf16, the situation improved slightly, but there is still current noise. I guess the problem is that the llm is around 80ms and the tts decode usage time is about 10ms. The generation speed cannot keep up with the transmission speed. May I ask if there are any methods to improve the generation speed?