Replay `recording-segments` endpoint deadlocks intermittently

[Cappsyhun]

Self-Hosted Version

26.4.1

CPU Architecture

x86_64

Docker Version

29.4.1

Docker Compose Version

v5.1.3

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Fresh Install

Steps to Reproduce

1. Fresh self-hosted Sentry 26.4.1 install with default config (filesystem filestore).

2. Set up a frontend with Sentry SDK and Replay enabled (e.g. sentry.javascript.angular@10.50.0).

3. Trigger an error to capture a replay (4 segments, ~4 MB total in our case).

4. Wait for ingest to complete (replay blobs visible at /data/files/30/<project_id>/<replay_id>/).

5. Call the recording-segments endpoint 5 times in quick succession:

   for i in 1 2 3 4 5; do
   curl -s -o /dev/null -w 'req %{url_effective}: code=%{http_code} time=%{time_total}s bytes=%{size_download}\n'
   --max-time 35
   -H "Authorization: Bearer $TOKEN"
   "$BASE/api/0/projects/$ORG/$PROJ/replays/$RID/recording-segments/?cursor=0%3A0%3A0&download=true&per_page=100"
   sleep 1
   done

6. Observe: most requests return code=200 but with bytes=1 and time=30s (nginx upstream timeout).
   The body contains only "[" — the streaming generator yields the opening bracket then hangs.

7. docker compose restart web temporarily resolves it (first 1-2 requests succeed),
   but the issue returns after a few requests.

Expected Result

The recording-segments endpoint should consistently return 200 OK with the full
streaming JSON payload (~4 MB in our test case) within ~100-200 ms, regardless
of how many requests have been made since the last web container restart.
The streaming generator should yield all segments after the opening "[" byte,
not hang after it.

Actual Result

After 1-2 successful requests post-restart, subsequent requests return 200 OK
but with only 1 byte body (just "[") after exactly 30 seconds — the nginx
upstream timeout. The body is truncated because the streaming generator hangs
after yielding the opening bracket. Every subsequent request behaves the same
way until "docker compose restart web" is run, after which the cycle repeats.

Event ID

No response

[Cappsyhun]

Replay recording-segments endpoint deadlocks intermittently in self-hosted 26.4.1 — ContextPropagatingThreadPoolExecutor issue

Summary

In a fresh self-hosted Sentry 26.4.1 install, the replay recording-segments endpoint intermittently hangs for 30+ seconds and returns only an opening [ byte to the client. The hang becomes consistent after several requests until the web container is restarted. The issue affects the playback path only — ingest, metadata storage, and blob storage all work correctly.

The root cause is in sentry/replays/usecases/reader.py iter_segment_data(), where the use of ContextPropagatingThreadPoolExecutor deadlocks under self-hosted conditions. Replacing it with a sequential for loop completely resolves the issue.

Environment

• Sentry version: 26.4.1 (sentry, version 26.4.1+e93c05476971a36729b28a08bb72abbfef82cf36)
• Deployment: Docker Compose self-hosted, single-node
• Host OS: Ubuntu 24, KVM virtualized, AMD EPYC host with host-passthrough CPU
• Resources: 32 GB RAM, ample CPU/disk; resource exhaustion ruled out (host idle, web container ~1.7 GB / 32 GB during repro)
• Filestore backend: filesystem (/data/files), default
• All services healthy per docker compose ps

Reproduction

1. Trigger an error in a Sentry-instrumented frontend that has Replay enabled (sentry.javascript.angular@10.50.0 in our case).
2. Wait for the replay to finish ingesting (4 segments in our test, ~4 MB total).
3. Call the recording-segments endpoint 5 times in quick succession with a valid auth token:

for i in 1 2 3 4 5; do
  curl -s -o /tmp/seg_$i.out -w 'code=%{http_code} time=%{time_total}s bytes=%{size_download}\n' \
    --max-time 35 \
    -H "Authorization: Bearer $TOKEN" \
    "$BASE/api/0/projects/$ORG/$PROJ/replays/$RID/recording-segments/?cursor=0%3A0%3A0&download=true&per_page=100"
  sleep 1
done

Observed (broken)

[req 1] code=200 time=30.061s bytes=1
[req 2] code=200 time=30.045s bytes=1
[req 3] code=200 time=0.111s  bytes=4244548   ← occasional success
[req 4] code=200 time=30.053s bytes=1
[req 5] code=200 time=30.031s bytes=1

The body is exactly [ — i.e. the streaming generator yields the opening bracket then hangs until nginx times out at 30s. The Sentry web view function logs the request as completed in ~130 ms (because the streaming generator is constructed and returned quickly), but the actual stream never produces further bytes.

After the workaround patch

[req 1] code=200 time=0.123s bytes=4244548
[req 2] code=200 time=0.109s bytes=4244548
[req 3] code=200 time=0.102s bytes=4244548

Consistent, fast, complete responses.

Diagnostics that ruled out other causes

I went through the full self-hosted stack to isolate the bug. Sharing in case it's useful for triage:

Storage layer is not the problem. Direct dd and Python open().read() on the underlying blob files complete in <1 ms each:

read 0: 848446 bytes in 0.46ms
read 1: 166929 bytes in 0.18ms
read 2: 519 bytes in 0.04ms
read 3: 340 bytes in 0.02ms

StorageBlob.get() is not the problem. Calling it directly via sentry exec returns each segment in ms:

seg 0: FOUND! (848446 bytes)
seg 1: FOUND! (166929 bytes)
seg 2: FOUND! (519 bytes)
seg 3: FOUND! (340 bytes)

fetch_segments_metadata() is not the problem. Returns 4 valid RecordingSegmentStorageMeta items in ~35 ms.

_download_segment() is not the problem. Each call returns in <5 ms with valid decompressed data:

seg 0: 4.38ms result OK (header=0b, body=2246061b)
seg 1: 3.54ms result OK (header=0b, body=1997164b)
seg 2: 0.39ms result OK (header=0b, body=557b)
seg 3: 0.32ms result OK (header=0b, body=761b)

Internal nginx logs the upstream timeout to web:9000:

nginx: upstream timed out (110: Operation timed out) while reading upstream,
request: "GET /api/0/projects/.../replays/.../recording-segments/..."

Worker thread states during hang: all worker threads show wchan=0 (running, not blocked on a syscall) — no futex_wait, no classic lock contention pattern. This is consistent with a context-propagation / generator issue rather than a lock deadlock.

Other components ruled out: Postgres (pgbouncer reachable, queries fast), Snuba/ClickHouse (replay metadata returns instantly), Redis, Memcached, Kafka — all healthy and responsive.

Root cause analysis

The hang is in sentry/replays/usecases/reader.py:

def iter_segment_data(
    segments: list[RecordingSegmentStorageMeta],
) -> Generator[tuple[int, memoryview]]:
    with ContextPropagatingThreadPoolExecutor(max_workers=10) as pool:
        segment_data = pool.map(_download_segment, segments)

    for i, result in enumerate(segment_data):
        if result is None:
            yield i, memoryview(b"[]")
        else:
            yield i, result[1]

Two suspicious aspects:

1. pool.map() returns a lazy iterator, but the with block exits (and pool.shutdown(wait=True) runs) before the iterator is consumed. The subsequent for loop then iterates a generator whose backing pool has been shut down. Whether this is the trigger or merely architectural smell, it's worth a closer look.

2. ContextPropagatingThreadPoolExecutor wraps the standard executor with sentry_sdk OpenTelemetry context propagation. Under self-hosted conditions (after several requests, even sequential ones), this appears to deadlock or produce a stalled generator that never yields. The web view's main response thread waits indefinitely on the iterator.

The hang is not present immediately after a web container restart — the first 1-2 requests may succeed — but accumulates over time. After ~5 requests, every subsequent request hangs for 30s and only restart resolves it.

Workaround

Replacing iter_segment_data with a sequential implementation eliminates the deadlock entirely:

def iter_segment_data(
    segments: list[RecordingSegmentStorageMeta],
) -> Generator[tuple[int, memoryview]]:
    # PATCHED: ContextPropagatingThreadPoolExecutor deadlocks in self-hosted.
    for i, segment in enumerate(segments):
        result = _download_segment(segment)
        if result is None:
            yield i, memoryview(b"[]")
        else:
            yield i, result[1]

For typical replay sizes (4 segments, ~4 MB total), the sequential version completes in ~10 ms total — fast enough that the loss of parallelism is invisible. For very large replays (hundreds of segments), parallelism may matter, but a non-broken parallel implementation would be preferable to the current one.

I applied this via sentry/enhance-image.sh to make it persistent across image rebuilds.

Suggested fix directions

A few possibilities the maintainers could consider, in increasing complexity:

1. Replace with sequential loop as in the workaround. Simple, correct, fast enough for typical workloads.
2. Materialize pool.map into a list inside the with block so the iterator is consumed before pool shutdown:

   with ContextPropagatingThreadPoolExecutor(max_workers=10) as pool:
       segment_data = list(pool.map(_download_segment, segments))
   for i, result in enumerate(segment_data):
       ...

   This addresses the lazy-iterator-after-shutdown concern, though it doesn't rule out a deeper ContextPropagatingThreadPoolExecutor bug.
3. Investigate ContextPropagatingThreadPoolExecutor for self-hosted-specific issues. The fact that it works on sentry.io SaaS but deadlocks here suggests something environment-specific (Python version, sentry_sdk version, OpenTelemetry config).

Additional notes

A separate, smaller issue I encountered during this debug that may also be worth tracking:

• After running ./install.sh post-CPU-fix on the host, the /data/files/30/ directory was created with root:root ownership instead of sentry:sentry. This caused ingest-replay-recordings to crash with PermissionError: [Errno 13] Permission denied: '/data/files/30/2/...' and silently drop all incoming replay blobs. A chown -R sentry:sentry /data/files/30 fixed it. Worth checking whether install.sh or the consumer's first-write should ensure correct ownership.

Happy to provide more diagnostic details or test fix candidates if useful.

[aminvakil]

Honestly I didn't read the wrong AI written comment completely, but it seems that this issue is not related to self-hosted but should be moved to sentry repository.