26.4.1: monitor_consumer wedges on idle-in-transaction after UPDATE sentry_monitorcheckin, check-in ingestion stops org-wide

[stumbaumr]

Self-Hosted Version

26.4.1

CPU Architecture

x86_64

Docker Version

28.0.4

Docker Compose Version

v5.1.3

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgraded from 26.3.1

Steps to Reproduce

1. Run 26.3.1 with ~50 active Crons monitors (mixed * * * * *, */5, */30, hourly).
2. Upgrade to 26.4.1 via the standard self-hosted install.
3. Monitor consumer processes a few dozen messages, then wedges.

Expected Result

Monitor check-ins continue to ingest normally after the upgrade.

Actual Result

Roughly 15–60 seconds after ingest-monitors starts processing post-upgrade, it silently wedges:

• Kafka consumer group ingest-monitors stays joined (valid consumer-id) but CURRENT-OFFSET stops advancing.
• Relay continues returning HTTP 202 at the edge; Kafka lag grows; check-ins never land in the monitor store.
• The healthcheck file /tmp/health.txt stops being touched, so Docker eventually marks the container unhealthy.
• No MAXPOLL until the configured interval expires.

Two recurring failure signatures (we've seen both in the same incident, not sure yet if they're two distinct bugs or two manifestations of one):

1. PG wedge. pg_stat_activity shows an open transaction idle in transaction / wait_event=ClientRead, last statement:
   UPDATE "sentry_monitorcheckin" SET "date_in_progress" = '...' WHERE "sentry_monitorcheckin"."id" = <id>
2. Row id differs each time. Immediately before silence the consumer logs:
   [INFO] sentry.utils.exceptions: No task state found in exception_grouping_context
3. Non-PG wedge. No activity on PG at all (no active/idle-in-tx consumer connections). The Python process is blocked somewhere not waiting on the database. Log stops mid-run after a routine check_in_closed message.

sentry-self-hosted-taskbroker-1 floods logs with:
WARN set_task_status: taskbroker::grpc::server: No pending activations
but this appears to be routine "nothing to do" noise, not an error.

Net effect: every active monitor flips to error with missed backfill; the system-wide Crons feature is effectively dead on the prod host.

Dev vs Prod comparison (this is the key data point)

Same 26.4.1 deployment on a second host (coreos-dev1-sentry1) handles check-ins fine — but that instance has 0 rows in sentry_monitorcheckin and near-zero organic traffic. Same images, same config, same override, same start.sh — the only difference is accumulated state on the broken host:

• sentry_monitorcheckin: 1,272,808 rows
• sentry_monitorincident: 2,013 rows
• sentry_monitorenvironment: 149 rows, 117 in status=4, 30 in status=5, 2 in status=0 post-upgrade

What we tried (none cleared the wedge)

1. docker compose restart ingest-monitors ingest-occurrences — wedges again within minutes.
2. kafka-consumer-groups --reset-offsets --to-latest for ingest-monitors and ingest-occurrences — gives temporary progress (seconds/minutes) before re-wedging at a different sentry_monitorcheckin.id.
3. --max-poll-interval-ms 300000 via docker-compose.override.yml — still wedges within the window.
4. --max-poll-interval-ms 1800000 — same, just longer wedge duration.
5. State surgery, in order of aggressiveness:
   - TRUNCATE sentry_monitorcheckin, sentry_monitorincident RESTART IDENTITY CASCADE (also cascades to sentry_monitorenvbrokendetection) → wedge reproduces.
   - Same truncate plus UPDATE sentry_monitorenvironment SET status=0, last_checkin=NULL, next_checkin=NULL, next_checkin_latest=NULL WHERE status != 1 to match dev's "fresh install" status distribution → still wedges.

Surgical cleanup of the tables directly referenced in the wedging UPDATE is not sufficient to restore function. State that triggers the bug evidently lives somewhere else (Redis cache? ClickHouse monitors_* tables? workflow-engine or issue-platform tables?), or the bug is independent of data and low-traffic dev simply doesn't hit it.

Event ID

N/A — this is self-hosted consumer-side; not user-facing event IDs.

[stumbaumr]

Update: deeper state surgery also fails

Since filing, we tried one more cleanup step beyond what was in the original report, and it still doesn't resolve the wedge.

What we additionally did (consumers stopped):

-- already done in the initial report                                                                                                                                                                                                                                                                                                                                             
TRUNCATE sentry_monitorcheckin, sentry_monitorincident RESTART IDENTITY CASCADE;
                                                                                                                                                                                                                                                                                                                                                                                  
-- NEW: also reset the env-level state to match a fresh install                                                                                                                                                                                                                                                                                                                   
UPDATE sentry_monitorenvironment                                                                                                                                                                                                                                                                                                                                                  
SET status = 0,                                                                                                                                                                                                                                                                                                                                                                   
    last_checkin = NULL,                                  
    next_checkin = NULL,
    next_checkin_latest = NULL
WHERE status != 1;   -- don't touch explicitly DISABLED ones                                                                                                                                                                                                                                                                                                                      

Verified row counts after surgery:

┌──────────────────────────────────┬───────────┬──────────────────────────────────────────────────┐                                                                                                                                                                                                                                                                               
│              Table               │   Rows    │                       Note                       │
├──────────────────────────────────┼───────────┼──────────────────────────────────────────────────┤
│ sentry_monitorcheckin            │ 0         │ truncated                                        │
├──────────────────────────────────┼───────────┼──────────────────────────────────────────────────┤
│ sentry_monitorincident           │ 0         │ truncated                                        │                                                                                                                                                                                                                                                                               
├──────────────────────────────────┼───────────┼──────────────────────────────────────────────────┤
│ sentry_monitorenvbrokendetection │ 0         │ cascaded                                         │                                                                                                                                                                                                                                                                               
├──────────────────────────────────┼───────────┼──────────────────────────────────────────────────┤                                                                                                                                                                                                                                                                               
│ sentry_monitorenvironment        │ 149       │ kept; 149 now status=0 (was 117× 4, 30× 5, 2× 0) │
├──────────────────────────────────┼───────────┼──────────────────────────────────────────────────┤                                                                                                                                                                                                                                                                               
│ sentry_monitor                   │ untouched │                                                  │
└──────────────────────────────────┴───────────┴──────────────────────────────────────────────────┘                                                                                                                                                                                                                                                                               

Then Kafka offsets reset to latest for both ingest-monitors and ingest-occurrences, consumers force-recreated with --max-poll-interval-ms 1800000.

Result: consumer joined, processed ~10–15 messages (monitors.consumer.check_in_closed status=1 updated_status=1), wedged again after ~15 s. Same pattern as before — UPDATE sentry_monitorcheckin SET date_in_progress = '…' WHERE id = left idle in transaction with wait_event=ClientRead.

Notable: the No task state found in exception_grouping_context log line does not appear before every wedge — some wedges occur with no Python-visible warning at all. So that log is a symptom of one code path, not a reliable marker.

Implication for root cause: the triggering state is not confined to the four monitor tables above. Candidates we haven't touched and that might hold it:

• Redis (Sentry's monitors-related cache / locks)
• ClickHouse monitors_* tables (Snuba side)
• sentry_groupopenperiod / workflow-engine tables — we saw sentry.models.groupopenperiod: Latest open period is not closed in ingest-occurrences logs right before its first wedge
• Or the bug is independent of stored state and only reliably triggered by non-trivial message volume (dev at ~0 msg/s doesn't hit it; prod at ~1–2 msg/s does)

[stumbaumr]

Root cause found — pymemcache 4.0.0 _recv() hangs under monitor-consumer concurrency

py-spy dump on the wedged /.venv/bin/python3 /.venv/bin/sentry run consumer ingest-monitors process shows this clearly.

Main thread — waiting on worker futures:                                                                                                                                                                                                                                                                                                                                          
wait (concurrent/futures/_base.py:305)                                                                                                                                                                                                                                                                                                                                            
wait (concurrent/futures/_base.py:659)                                                                                                                                                                                                                                                                                                                                            
process_batch (sentry/monitors/consumers/monitor_consumer.py:1101)                                                                                                                                                                                                                                                                                                                
submit (arroyo/processing/strategies/run_task.py:68)                                                                                                                                                                                                                                                                                                                              
__flush (arroyo/processing/strategies/buffer.py:125)                                                                                                                                                                                                                                                                                                                              
poll (arroyo/processing/strategies/buffer.py:165 → reduce.py:110 → batching.py:82 → healthcheck.py:32)                                                                                                                                                                                                                                                                            
_run_once (arroyo/processing/processor.py:464)                                                                                                                                                                                                                                                                                                                                    

Every ThreadPoolExecutor-1_N worker (N≥2) — blocked in pymemcache _recv():

_recv (pymemcache/client/base.py:1750)                                                                                                                                                                                                                                                                                                                                            
_readline (pymemcache/client/base.py:1658)                                                                                                                                                                                                                                                                                                                                        
_fetch_cmd (…:1145)    # or _misc_cmd (…:1291) for delete                                                                                                                                                                                                                                                                                                                         
get / delete (pymemcache/client/hash.py:347 / 423)                                                                                                                                                                                                                                                                                                                                
get / set (django/core/cache/backends/memcached.py:75 / 82)                                                                                                                                                                                                                                                                                                                       
get_from_cache / __post_save (sentry/db/models/manager/base.py:319 / 215)                                                                                                                                                                                                                                                                                                         
_process_checkin (sentry/monitors/consumers/monitor_consumer.py:486)                                                                                                                                                                                                                                                                                                              
process_checkin (monitor_consumer.py:1033)                                                                                                                                                                                                                                                                                                                                        
process_checkin_group (monitor_consumer.py:1046)                                                                                                                                                                                                                                                                                                                                  
wrapped_fn (sentry_sdk/integrations/threading.py:172)                                                                                                                                                                                                                                                                                                                             

Kernel / socket state at the time of the wedge:

• All blocked threads are in recvfrom() (syscall 45).
• TCP sockets to memcached:11211 show State=ESTAB, Recv-Q=0, Send-Q=0 — memcached has no pending bytes to deliver and nothing was stuck waiting to send. The client is reading past the end of a completed response.
• Memcached server is healthy: stats responds instantly, curr_connections=75 / max_connections=1024, rejected_connections=0. An independent get from another client returns END in milliseconds.

Versions involved:

• Sentry self-hosted 26.4.1
• pymemcache==4.0.0
• memcached 1.6.26
• Django cache backend: django.core.cache.backends.memcached.PyMemcacheCache
• Default CACHES['default']['OPTIONS'] before fix: {'ignore_exc': True} — no socket timeout set. With ignore_exc but no timeout, recv() blocks forever and no exception is ever raised for Django to swallow.

Why 26.4.1 and not 26.3.1

The wedge happens under concurrency: _process_checkin runs in a ThreadPoolExecutor, and each worker issues pymemcache gets/sets via get_from_cache + __post_save. The pymemcache bug apparently needs concurrent operations on the same (or multiple) connections. If 26.4.1 introduced the ThreadPoolExecutor path in process_checkin_group, or increased its fan-out, that would
explain the regression timing. (The stack unambiguously shows it is in use.)

Why dev didn't show it

Our dev host runs the same 26.4.1 image + same config + same docker-compose.override.yml, but has ~zero check-in traffic, so the worker pool never fans out enough to trip the bug.

Workaround that clears the production wedge

Append to sentry/sentry.conf.py:

CACHES['default']['OPTIONS'].update({'timeout': 5, 'connect_timeout': 3})

After applying and recreating ingest-monitors + ingest-occurrences:

• Consumer catches up immediately (lag 0 on ingest-monitors).
• Bursts of 15+ check_in_closed messages per second processed.
• idle in transaction durations now measured in tens of ms (was growing to full poll-interval).

What upstream should consider

1. Default a reasonable socket timeout in pymemcache usage (Sentry-side) so ignore_exc=True can actually recover. Currently the two options together are misleading — without a timeout, nothing makes the client raise.
2. The underlying pymemcache _recv() loop after a completed response is worth a deeper look.
3. Self-hosted clusters upgrading from 26.3.x → 26.4.1 with any real check-in volume will hit this immediately.

Happy to run any further diagnostics if that helps.

[jsunier]

Workaround that clears the production wedge

Append to sentry/sentry.conf.py:

CACHES['default']['OPTIONS'].update({'timeout': 5, 'connect_timeout': 3})

After applying and recreating ingest-monitors + ingest-occurrences:

* Consumer catches up immediately (lag 0 on ingest-monitors).

* Bursts of 15+ check_in_closed messages per second processed.

* idle in transaction durations now measured in tens of ms (was growing to full poll-interval).

Thanks @stumbaumr, i can confirm it fixed the issue on our side by simply changing sentry.conf.py like this :

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
        "LOCATION": ["memcached:11211"],
        "TIMEOUT": 3600,
-		"OPTIONS": {"ignore_exc": True},
+		"OPTIONS": {"ignore_exc": True, 'timeout': 5, 'connect_timeout': 3},
    }
}

then rerunning the install.sh script to rebuild the images and restarting the docker compose stack 👍

[aldy505]

Hi! I see that you're able to find a solution for this. The observation is great.

What upstream should consider

1. Default a reasonable socket timeout in pymemcache usage (Sentry-side) so ignore_exc=True can actually recover. Currently the two options together are misleading — without a timeout, nothing makes the client raise.
2. The underlying pymemcache _recv() loop after a completed response is worth a deeper look.
3. Self-hosted clusters upgrading from 26.3.x → 26.4.1 with any real check-in volume will hit this immediately.

This looks like something that should be on our documentation. It'd be great if you can submit a pull request for this.