From 6c9aa2adaaa3a5fc5eab0b56c1c9cdcdf124bd06 Mon Sep 17 00:00:00 2001 From: jaderf-sm Date: Wed, 18 Feb 2026 08:03:28 +0000 Subject: [PATCH 1/3] Add raw websocket documentation --- .../realtime-transcription-websocket.mdx | 4 + .../realtime/guides/websocket-protocol.mdx | 337 ++++++++++++++++++ docs/speech-to-text/realtime/input.mdx | 7 +- docs/speech-to-text/realtime/quickstart.mdx | 4 + 4 files changed, 349 insertions(+), 3 deletions(-) create mode 100644 docs/speech-to-text/realtime/guides/websocket-protocol.mdx diff --git a/docs/api-ref/realtime-transcription-websocket.mdx b/docs/api-ref/realtime-transcription-websocket.mdx index b8e6da37..cbda589d 100644 --- a/docs/api-ref/realtime-transcription-websocket.mdx +++ b/docs/api-ref/realtime-transcription-websocket.mdx @@ -52,6 +52,10 @@ sequenceDiagram API-->>Client: EndOfTranscript ``` +:::tip +For a step-by-step walkthrough of this protocol, see the [WebSocket protocol guide](/speech-to-text/realtime/guides/websocket-protocol). +::: + :::warning #### Browser based transcription diff --git a/docs/speech-to-text/realtime/guides/websocket-protocol.mdx b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx new file mode 100644 index 00000000..2955bd5b --- /dev/null +++ b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx @@ -0,0 +1,337 @@ +--- +sidebar_label: 'WebSocket protocol' +sidebar_position: 0 +description: 'Connect to the Speechmatics Realtime API using raw WebSocket messages.' +keywords: [websocket, protocol, realtime, raw, go, rust, java, transcription, speech recognition, asr] +--- + +import { LinkCard } from "@site/src/theme/LinkCard"; +import { Grid } from "@radix-ui/themes"; +import { BookOpen, ChevronsRight } from "lucide-react"; + +# WebSocket protocol guide + +This guide walks through the raw WebSocket message flow for the Speechmatics Realtime API. It is intended for developers who want to understand the protocol before using one of the available SDKs, or who are building a client in a language without an official SDK (Go, Rust, Java, etc.). + +For SDK-based quickstarts, see the [Realtime quickstart](/speech-to-text/realtime/quickstart). + +## Prerequisites + +- A Speechmatics API key — [create one here](https://portal.speechmatics.com/settings/api-keys) +- A WebSocket client capable of sending both JSON text frames and binary frames +- A sample audio file — see [Prepare a sample audio file](#prepare-a-sample-audio-file) below + +:::info +This guide shows the raw WebSocket messages themselves, not tool-specific commands. You can use any WebSocket library or client in your preferred language. +::: + +## Protocol overview + +The Realtime API uses a WebSocket connection. Control messages (session setup, acknowledgements, transcripts) are JSON text frames. Audio data is sent as binary frames. + +```mermaid +sequenceDiagram + participant Client + participant Server + Client->>Server: WebSocket upgrade with auth + Server-->>Client: 101 Switching Protocols + Client->>Server: StartRecognition (JSON) + Server-->>Client: RecognitionStarted (JSON) + loop Audio streaming + Client->>Server: Audio data (binary) + Server-->>Client: AudioAdded (JSON) + Server-->>Client: AddTranscript (JSON) + end + Client->>Server: EndOfStream (JSON) + Server-->>Client: EndOfTranscript (JSON) +``` + +## Connect + +{/* */} + +### Endpoint + +Connect to the Realtime WebSocket API at: + +``` +wss://eu.rt.speechmatics.com/v2 +``` + +A US endpoint is also available at `wss://us.rt.speechmatics.com/v2`. See the [full endpoint list](/get-started/authentication#supported-endpoints) for all regions. + +{/* */} + +### Authentication + +**Server-side:** Pass your API key as a Bearer token in the WebSocket upgrade request headers: + +``` +GET /v2 HTTP/1.1 +Host: eu.rt.speechmatics.com +Upgrade: websocket +Connection: Upgrade +Authorization: Bearer YOUR_API_KEY +``` + +**Browser-side:** Browsers cannot set custom headers on WebSocket connections. Use a [temporary key](/get-started/authentication#temporary-keys) passed as a query parameter instead: + +``` +wss://eu.rt.speechmatics.com/v2?jwt=TEMPORARY_KEY +``` + +To generate a temporary key: + +```bash +curl -L -X POST "https://mp.speechmatics.com/v1/api_keys?type=rt" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $API_KEY" \ + -d '{"ttl": 60}' +``` + +The response contains a `key_value` field with your temporary key, valid for the specified TTL (in seconds). + +### Successful handshake + +A successful WebSocket upgrade returns: + +``` +HTTP/1.1 101 Switching Protocols +Connection: upgrade +Upgrade: WebSocket +``` + +If the handshake fails, you'll receive one of: +- `400 Bad Request` — malformed request +- `401 Unauthorized` — invalid API key +- `405 Method Not Allowed` — non-GET request method + +## Message lifecycle + +### 1. Start the session + +After the WebSocket connection is open, send a `StartRecognition` JSON message to begin transcription. + +**Minimal example (raw PCM audio):** + +```json +{ + "message": "StartRecognition", + "audio_format": { + "type": "raw", + "encoding": "pcm_s16le", + "sample_rate": 16000 + }, + "transcription_config": { + "language": "en" + } +} +``` + +**With common options:** + +```json +{ + "message": "StartRecognition", + "audio_format": { + "type": "raw", + "encoding": "pcm_s16le", + "sample_rate": 16000 + }, + "transcription_config": { + "language": "en", + "operating_point": "enhanced", + "max_delay": 2.0, + "enable_partials": true + } +} +``` + +#### `audio_format` fields + +| Field | Description | +|-------|-------------| +| `type` | `"raw"` for raw PCM audio, or `"file"` for container formats (wav, mp3, ogg, flac, etc.) | +| `encoding` | Required when `type` is `"raw"`. One of: `pcm_s16le`, `pcm_f32le`, `mulaw` | +| `sample_rate` | Required when `type` is `"raw"`. Sample rate in Hz (e.g., `16000`, `44100`) | + +When using `"type": "file"`, the server detects the encoding and sample rate from the file headers. No `encoding` or `sample_rate` fields are needed: + +```json +{ + "audio_format": { "type": "file" } +} +``` + +See the [API reference: StartRecognition](/api-ref/realtime-transcription-websocket#startrecognition) for the full schema including translation and audio events configuration. + +### 2. Receive session confirmation + +The server responds with a `RecognitionStarted` message: + +```json +{ + "message": "RecognitionStarted", + "id": "807670e9-14af-4fa2-9e8f-5d525c22156e", + "language_pack_info": { + "adapted": false, + "itn": true, + "language_description": "English", + "word_delimiter": " ", + "writing_direction": "left-to-right" + } +} +``` + +The `id` field is the unique session identifier. Do not send audio until you receive this message. + +:::info +The server may also send `Info` messages immediately after the handshake, reporting recognition quality and concurrent session usage. These are informational and do not require a response. +::: + +### 3. Stream audio data + +Send audio as **binary WebSocket frames**. Each frame contains raw audio bytes — there is no JSON wrapper. A chunk size of around 4096 bytes works well. + +The server acknowledges each audio chunk with an `AudioAdded` JSON message: + +```json +{ + "message": "AudioAdded", + "seq_no": 1 +} +``` + +The `seq_no` increments with each chunk received. Keep track of the latest `seq_no` — you'll need it when ending the session. + +:::info +**Backpressure:** If you send audio faster than real-time, monitor the `AudioAdded` acknowledgements. Avoid having more than a few hundred unacknowledged chunks in flight. If the server's buffer fills, the connection may close. +::: + +### 4. Receive transcripts + +As audio is processed, the server sends `AddTranscript` messages with finalized transcription results: + +```json +{ + "message": "AddTranscript", + "format": "2.9", + "metadata": { + "start_time": 0.54, + "end_time": 2.1, + "transcript": "Hello world." + }, + "results": [ + { + "type": "word", + "start_time": 0.54, + "end_time": 0.96, + "alternatives": [ + { "content": "Hello", "confidence": 0.99 } + ] + }, + { + "type": "word", + "start_time": 1.02, + "end_time": 1.5, + "alternatives": [ + { "content": "world", "confidence": 0.98 } + ] + }, + { + "type": "punctuation", + "start_time": 1.5, + "end_time": 1.5, + "alternatives": [ + { "content": ".", "confidence": 1.0 } + ] + } + ] +} +``` + +The `metadata.transcript` field contains the pre-formatted text. The `results` array provides word-level timing and confidence scores. + +If you enabled `enable_partials` in your `transcription_config`, you'll also receive `AddPartialTranscript` messages. These have the same structure but may be revised as more audio is processed. Use partials for low-latency display, then replace them with finals as they arrive. + +See the [output documentation](/speech-to-text/realtime/output) for details on latency tuning with `max_delay`. + +### 5. End the session + +When you've finished sending audio, send an `EndOfStream` message. Set `last_seq_no` to the `seq_no` from the last `AudioAdded` message you received: + +```json +{ + "message": "EndOfStream", + "last_seq_no": 42 +} +``` + +The server will finalize any remaining transcripts and respond with: + +```json +{ + "message": "EndOfTranscript" +} +``` + +After receiving `EndOfTranscript`, close the WebSocket connection. + +## Prepare a sample audio file + +To test the protocol, you need an audio file to stream. + +**Option A: Use a pre-recorded file** + +Download the [example.wav](https://github.com/speechmatics/speechmatics-js-sdk/raw/7d219bfee9166736e6aa21598535a194387b84be/examples/nodejs/example.wav) sample file. When using a `.wav` file, set `audio_format` to `{ "type": "file" }` in your `StartRecognition` message. + +{/* */} + +**Option B: Generate raw PCM audio** + +Convert any audio file to raw PCM using `ffmpeg`: + +```bash +ffmpeg -i input.wav -f s16le -acodec pcm_s16le -ar 16000 -ac 1 output.raw +``` + +When using raw PCM, set `audio_format` to: + +```json +{ "type": "raw", "encoding": "pcm_s16le", "sample_rate": 16000 } +``` + +## Error handling + +If an error occurs, the server sends an `Error` JSON message followed by a WebSocket close. Common close codes: + +| Close code | Payload | Description | +|-----------|---------|-------------| +| 1003 | `protocol_error` | Malformed or unexpected message | +| 1011 | `internal_error` | Server error | +| 4001 | `not_authorised` | Invalid API key | +| 4005 | `quota_exceeded` | Maximum concurrent connections reached | +| 4013 | `job_error` | Server unable to process the job | + +:::tip +For `4005 quota_exceeded`, `4013 job_error`, and `1011 internal_error`, wait 5-10 seconds before retrying. +::: + +See the [API reference: WebSocket errors](/api-ref/realtime-transcription-websocket#websocket-errors) for the full list of close codes. + +## Next steps + + + } + /> + } + /> + diff --git a/docs/speech-to-text/realtime/input.mdx b/docs/speech-to-text/realtime/input.mdx index 01e3a3f0..93b93c95 100644 --- a/docs/speech-to-text/realtime/input.mdx +++ b/docs/speech-to-text/realtime/input.mdx @@ -31,6 +31,7 @@ After receiving a `RecognitionStarted` message, you can start sending audio over ## Next steps -View our guides: -- [using a microphone](docs/speech-to-text/realtime/guides/python-using-microphone.mdx) to learn how to capture audio from a microphone. -- [using FFMPEG](docs/speech-to-text/realtime/guides/python-using-ffmpeg.mdx) to find out how to pipe microphone audio to the API. \ No newline at end of file +View our guides: +- [using a microphone](docs/speech-to-text/realtime/guides/python-using-microphone.mdx) to learn how to capture audio from a microphone. +- [using FFMPEG](docs/speech-to-text/realtime/guides/python-using-ffmpeg.mdx) to find out how to pipe microphone audio to the API. +- [WebSocket protocol guide](/speech-to-text/realtime/guides/websocket-protocol) for a step-by-step protocol walkthrough. \ No newline at end of file diff --git a/docs/speech-to-text/realtime/quickstart.mdx b/docs/speech-to-text/realtime/quickstart.mdx index b131207a..f9487809 100644 --- a/docs/speech-to-text/realtime/quickstart.mdx +++ b/docs/speech-to-text/realtime/quickstart.mdx @@ -16,6 +16,10 @@ import pythonRadioExample from "./assets/url-example.py?raw" The easiest way to try Realtime transcription is via the [web portal](https://portal.speechmatics.com/jobs/create/real-time). ::: +:::info +Looking for the raw WebSocket protocol? See our [WebSocket protocol guide](/speech-to-text/realtime/guides/websocket-protocol) for a language-agnostic walkthrough. +::: + ## Using the Realtime SaaS webSocket API ### 1. Create an API key From ac70d6e43aefa2c0824b3958fb53100fd3934998 Mon Sep 17 00:00:00 2001 From: jaderf-sm Date: Wed, 18 Feb 2026 08:16:23 +0000 Subject: [PATCH 2/3] Add information on Backpressure --- .../realtime/guides/websocket-protocol.mdx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/speech-to-text/realtime/guides/websocket-protocol.mdx b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx index 2955bd5b..8217620e 100644 --- a/docs/speech-to-text/realtime/guides/websocket-protocol.mdx +++ b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx @@ -204,8 +204,14 @@ The server acknowledges each audio chunk with an `AudioAdded` JSON message: The `seq_no` increments with each chunk received. Keep track of the latest `seq_no` — you'll need it when ending the session. -:::info -**Backpressure:** If you send audio faster than real-time, monitor the `AudioAdded` acknowledgements. Avoid having more than a few hundred unacknowledged chunks in flight. If the server's buffer fills, the connection may close. +#### Backpressure + +If you send audio faster than real-time, unacknowledged chunks can pile up. The server buffers incoming audio internally, but if that buffer fills, TCP-level buffers will also fill, eventually causing write failures and connection closure. + +To avoid this, track how many chunks you've sent versus how many `AudioAdded` acknowledgements you've received. If the gap grows too large, pause sending until the server catches up. The Python SDK caps this at 512 outstanding unacknowledged chunks, with a 120-second timeout waiting for an acknowledgement before giving up. These are reasonable starting points for your own implementation, but the exact values are an implementation choice. + +:::tip +A simple approach: maintain a counter that increments when you send a chunk and decrements when you receive an `AudioAdded`. If the counter exceeds your chosen limit, wait before sending the next chunk. ::: ### 4. Receive transcripts From c9e3bbb632a91527a6ba4ec36978d5511f6b57ef Mon Sep 17 00:00:00 2001 From: jaderf-sm Date: Wed, 18 Feb 2026 08:23:07 +0000 Subject: [PATCH 3/3] Websocket: Add section on limits --- .../realtime/guides/websocket-protocol.mdx | 36 ++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/docs/speech-to-text/realtime/guides/websocket-protocol.mdx b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx index 8217620e..09c041a0 100644 --- a/docs/speech-to-text/realtime/guides/websocket-protocol.mdx +++ b/docs/speech-to-text/realtime/guides/websocket-protocol.mdx @@ -7,7 +7,7 @@ keywords: [websocket, protocol, realtime, raw, go, rust, java, transcription, sp import { LinkCard } from "@site/src/theme/LinkCard"; import { Grid } from "@radix-ui/themes"; -import { BookOpen, ChevronsRight } from "lucide-react"; +import { BookOpen, ChevronsRight, Gauge } from "lucide-react"; # WebSocket protocol guide @@ -307,6 +307,33 @@ When using raw PCM, set `audio_format` to: { "type": "raw", "encoding": "pcm_s16le", "sample_rate": 16000 } ``` +## Session limits + +The server will automatically end a session if any of these limits are reached: + +| Limit | Threshold | Warning type | +|-------|-----------|-------------| +| Maximum session duration | 48 hours | `session_timeout` | +| No audio data sent | 1 hour | `idle_timeout` | +| No audio or ping/pongs | 3 minutes | — | + +Before auto-termination, the server sends `Warning` messages to give your client time to react. For example: + +```json +{ + "message": "Warning", + "type": "idle_timeout", + "reason": "Idle timeout approaching. No audio received for 45 minutes." +} +``` + +- **`idle_timeout` warnings** are sent at 15, 10, and 5 minutes before the 1-hour limit. +- **`session_timeout` warnings** are sent at 45, 30, and 15 minutes before the 48-hour limit. + +When a session is auto-terminated, the server sends an in-band `Error` message followed by a WebSocket close with code `1008`. For long-running sessions, handle these warnings gracefully — you can disconnect and immediately reconnect a new session with minimal disruption. + +See [Limits](/speech-to-text/realtime/limits) for monthly usage quotas and concurrency limits. + ## Error handling If an error occurs, the server sends an `Error` JSON message followed by a WebSocket close. Common close codes: @@ -314,6 +341,7 @@ If an error occurs, the server sends an `Error` JSON message followed by a WebSo | Close code | Payload | Description | |-----------|---------|-------------| | 1003 | `protocol_error` | Malformed or unexpected message | +| 1008 | `policy_violation` | Session limit exceeded (idle or duration timeout) | | 1011 | `internal_error` | Server error | | 4001 | `not_authorised` | Invalid API key | | 4005 | `quota_exceeded` | Maximum concurrent connections reached | @@ -340,4 +368,10 @@ See the [API reference: WebSocket errors](/api-ref/realtime-transcription-websoc href="/speech-to-text/realtime/quickstart" icon={} /> + } + />