[Feature] FunASR as self-hosted speech-to-text tool for voice agents

[LauraGPT]

Summary

For voice-enabled multi-agent systems, agents need a reliable STT backend. FunASR (16K+ stars) now offers an OpenAI-compatible transcription API that AutoGen agents can use directly.

Proposed Integration

FunASR provides examples/openai_api/server.py — a FastAPI server that exposes /v1/audio/transcriptions (same format as OpenAI). Any AutoGen agent using OpenAI audio can switch to FunASR by changing the base_url:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

# Use in agent tool
def transcribe_audio(file_path: str) -> str:
    result = client.audio.transcriptions.create(
        model="sensevoice", file=open(file_path, "rb")
    )
    return result.text

Advantages over OpenAI Whisper API

	FunASR	OpenAI Whisper API
Speed	170x realtime	~1x realtime
Cost	Free (self-hosted)	$0.006/min
Privacy	Local, no data leaves	Cloud
Speaker ID	Built-in	No
Languages	50+	57

Setup

pip install funasr fastapi uvicorn python-multipart
python examples/openai_api/server.py --device cuda

Repo: https://github.com/modelscope/FunASR

[scosemicolon]

A practical integration path could be to start with an example rather than a provider abstraction: show an AutoGen voice agent using an OpenAI-compatible transcription base URL, with FunASR as the local server.

That keeps the first step low-risk because users can swap only the transcription client configuration:

• same request shape as OpenAI transcription
• explicit local base_url
• clear note that model names and language options are FunASR-specific
• fallback behavior when the local service is unavailable

For production voice agents, I would also document two operational checks in the example: audio sample-rate normalization before sending to FunASR, and a timeout around transcription calls so one stalled local inference request does not block the whole multi-agent conversation loop.

[LauraGPT]

Exactly right — FunASR already exposes an OpenAI-compatible endpoint, so the integration is just a base_url swap:

# Start local FunASR server
pip install funasr vllm fastapi uvicorn python-multipart
funasr-server --device cuda

# In AutoGen agent config — same API shape as OpenAI
import openai

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

with open("audio.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="sensevoice",  # or "paraformer", "fun-asr-nano"
        file=f,
        response_format="verbose_json"  # includes timestamps + speaker IDs
    )
print(transcript.text)

Good callout on the timeout — funasr-server processes most segments in <100ms (170x realtime), but a timeout is still wise for resilience. Sample rate normalization is handled server-side (auto-resamples to 16kHz).

Happy to contribute an example to the AutoGen examples directory if maintainers are interested.