Canary-v2 streamatt speech processor

[azziko]

Hello,

I implemented a support for decoding_input_ids and tested it to see if it works. Although it's not merged yet, I'm working towards contributing canary with streamatt to the repo.

I have a question about the implementation of audio history. I see from the base_streamatt implementation that the audio is supposed to be stored in mel-features. Although it is possible to extract the mel-features first in NeMo framework, it's much easier to work with raw waveform history. I was thinking of some options how to tweak the implementation so that the raw history update is supported:

1. In self.audio_subsampling_factor put subsampling_factor * MEL_HOP_SAMPLES. This maps one encoder frame -> raw form. I was worried this might mess up the semantics of the code a bit.
2. The second option would be to override _update_audio_history completely, but then there is repeating code.

How would you see this implemented best?

[mgaido91]

Thank you very much @azziko ! Honestly, I think that if possible working with mel features would be preferred for a couple of reasons:

1. When you decide how much audio to retain with the attention matrix, you do this selection on the network input (ie. mel features). If you want to keep the waveform you have to map this back to the waveform.
2. Extracting mel features, although not the most expesive thing, has some compute overhead, which is not negligible. So if we can avoid doing it multiple times, the whole thing is a bit more efficient.

Thank you!