Skip to content

feat: added new eval set + default behaviour fixes #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VarunGitGood merged 1 commit into from
Jun 23, 2026
Merged

feat: added new eval set + default behaviour fixes #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
{
"query": "why are users getting disconnected from the gateway",
"expected_clarification": {
"should_clarify": false,
"rationale": "'gateway' is a registered service and the time window (2026-06-10 02:14–02:48) is unambiguous in the logs."
},
"expected_answer": {
"incident_window": {
"start": "2026-06-10T02:14:00Z",
"end": "2026-06-10T02:48:00Z"
},
"affected_services": ["read-states-svc", "gateway", "message-svc", "cdn-edge"],
"trigger_event": {
"service": "read-states-svc",
"timestamp_approx": "2026-06-10T02:14:52Z",
"log_line_must_contain": "Cassandra timeout"
},
"propagation_chain_must_include_in_order": [
{
"service": "read-states-svc",
"ts_approx": "2026-06-10T02:14:52Z",
"what_must_mention": "Cassandra / BATCH APPLY / OperationTimedOutException"
},
{
"service": "gateway",
"ts_approx": "2026-06-10T02:16:15Z",
"what_must_mention": "read_states / mark_read / timeout / disconnect"
},
{
"service": "gateway",
"ts_approx": "2026-06-10T02:17:45Z",
"what_must_mention": "reconnect storm"
},
{
"service": "message-svc",
"ts_approx": "2026-06-10T02:19:05Z",
"what_must_mention": "thread pool / saturated / 512"
},
{
"service": "cdn-edge",
"ts_approx": "2026-06-10T02:16:44Z",
"what_must_mention": "502"
}
],
"root_cause_must_mention": [
"Cassandra",
"read-states",
"timeout",
"reconnect",
"pool"
],
"ruled_out_hypotheses_must_include": [
{
"hypothesis_about": "presence-svc",
"rationale": "presence-svc shows elevated activity caused by the gateway reconnect storm, not a cause of it — no Cassandra errors, no connection pool exhaustion"
}
],
"assumptions_must_echo": [
"'gateway' refers to the WebSocket gateway service"
],
"confidence": "high",
"gaps_must_be_empty_or_minor": true
},
"common_failure_modes_to_grade_against": [
"Names gateway as root cause (it is the dominant symptom, not the trigger). Should be downgraded.",
"Misses read-states-svc entirely because it was not mentioned in the query.",
"Includes presence-svc in affected_services without noting it is a downstream symptom of reconnects.",
"Fails to link Cassandra partition leader re-election → read_states timeouts → WebSocket disconnects → reconnect storm chain.",
"Skips cdn-edge 502s as evidence of the gateway impact radius.",
"Does not identify OperationTimedOutException as the trigger mechanism."
],
"what_this_dataset_tests": [
"Multi-hop causal chain: 4 services deep from trigger (read-states) to visible symptom (cdn-edge 502).",
"Cross-service auto-sweep: read-states-svc errors must surface even though only 'gateway' was queried.",
"Red-herring elimination: presence-svc elevated but is a symptom of the reconnect storm, not a cause.",
"Infrastructure-level root cause: Cassandra partition leader re-election, not application code.",
"Reconnect amplification pattern: disconnects → reconnects → message-svc saturation loop."
]
}
30 changes: 30 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/live/cdn-edge.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
2026-06-10T02:10:00Z [INFO] CDN edge node cdn-edge-01 started (PoP: us-east-1)
2026-06-10T02:10:01Z [INFO] Origin backend: gateway-cluster.internal (4 instances, health checked)
2026-06-10T02:11:00Z [INFO] Health check ok — origin P99: 18ms, cache hit rate: 96%
2026-06-10T02:12:00Z [INFO] Health check ok — origin P99: 19ms, cache hit rate: 96%
2026-06-10T02:13:00Z [INFO] Health check ok — origin P99: 21ms, cache hit rate: 96%
2026-06-10T02:14:00Z [INFO] Health check ok — origin P99: 20ms, cache hit rate: 96%
2026-06-10T02:15:00Z [INFO] Health check ok — origin P99: 148ms, cache hit rate: 96%
2026-06-10T02:16:00Z [INFO] Health check ok — origin P99: 892ms, cache hit rate: 95%
2026-06-10T02:16:44Z [ERROR] Origin 502: gateway-01.internal /api/v9/channels/8821/messages returned 502 Bad Gateway
2026-06-10T02:16:46Z [ERROR] Origin 502: gateway-02.internal /api/v9/channels/4491/messages returned 502 Bad Gateway
2026-06-10T02:16:50Z [ERROR] Origin 502: gateway-01.internal /gateway?v=9&encoding=json returned 502 Bad Gateway
2026-06-10T02:16:55Z [ERROR] Origin 502: gateway-03.internal /api/v9/users/@me/channels returned 502 Bad Gateway
2026-06-10T02:16:58Z [ERROR] Origin 502 rate: 24% of requests to gateway cluster
2026-06-10T02:17:10Z [ERROR] Origin 502 rate: 41% of requests to gateway cluster
2026-06-10T02:17:20Z [ERROR] Origin 502 rate: 58% of requests to gateway cluster — circuit breaker threshold approaching
2026-06-10T02:17:30Z [WARNING] Serving stale cache for /api/v9/channels/* (TTL extended to 300s due to origin degradation)
2026-06-10T02:18:00Z [ERROR] Origin 502 rate: 62% of requests — gateway cluster under severe load
2026-06-10T02:18:30Z [WARNING] Edge cache miss rate elevated: 34% (baseline 4%) — reconnecting clients bypass cache with fresh IDENTIFY requests
2026-06-10T02:19:00Z [ERROR] Origin 502 rate: 71% — gateway message-svc pool exhausted, fanout requests failing
2026-06-10T02:20:00Z [ERROR] Origin 502 rate: 69%
2026-06-10T02:22:00Z [ERROR] Origin 502 rate: 65%
2026-06-10T02:25:00Z [ERROR] Origin 502 rate: 58%
2026-06-10T02:30:00Z [WARNING] Origin 502 rate: 44% — slowly improving
2026-06-10T02:35:00Z [WARNING] Origin 502 rate: 28%
2026-06-10T02:38:00Z [WARNING] Origin 502 rate: 12% — gateway recovering
2026-06-10T02:40:00Z [WARNING] Origin 502 rate: 4%
2026-06-10T02:42:10Z [INFO] Origin 502 rate dropped to 0 — gateway cluster healthy
2026-06-10T02:43:00Z [INFO] Health check ok — origin P99: 22ms, cache hit rate: 94%
2026-06-10T02:45:00Z [INFO] Health check ok — origin P99: 20ms, cache hit rate: 95%
2026-06-10T02:48:00Z [INFO] Edge health nominal — origin P99: 18ms, cache hit rate: 96%
46 changes: 46 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/live/gateway.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
2026-06-10T02:10:00Z [INFO] Gateway started, version 8.14.2
2026-06-10T02:10:01Z [INFO] WebSocket listener bound on :443 (TLS termination via envoy)
2026-06-10T02:10:02Z [INFO] Connected clients: 84312
2026-06-10T02:11:00Z [INFO] Health check ok — connected clients: 85102, read_states P99: 8ms
2026-06-10T02:12:00Z [INFO] Health check ok — connected clients: 85318, read_states P99: 9ms
2026-06-10T02:13:00Z [INFO] Health check ok — connected clients: 85541, read_states P99: 11ms
2026-06-10T02:14:00Z [INFO] Health check ok — connected clients: 85749, read_states P99: 10ms
2026-06-10T02:15:00Z [INFO] Health check ok — connected clients: 86001, read_states P99: 312ms
2026-06-10T02:15:30Z [WARNING] read_states mark_read API P99 latency: 3214ms (threshold 200ms) — upstream degraded
2026-06-10T02:15:45Z [WARNING] read_states mark_read API P99 latency: 5800ms — requests queuing
2026-06-10T02:16:00Z [WARNING] mark_read upstream timeout rate: 12% of requests timing out
2026-06-10T02:16:15Z [ERROR] read_states mark_read timeout for session_id=sess_4f9a2b after 5000ms
2026-06-10T02:16:18Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_4f9a2b user_id=u_9f2a1c
2026-06-10T02:16:22Z [ERROR] read_states mark_read timeout for session_id=sess_7c1e3d after 5000ms
2026-06-10T02:16:24Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_7c1e3d user_id=u_3b8e2f
2026-06-10T02:16:30Z [ERROR] mark_read upstream timeout rate: 68% of requests timing out
2026-06-10T02:16:45Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_2a9c4e user_id=u_f7a1d9
2026-06-10T02:16:50Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_b3f1c2 user_id=u_e2c4a1
2026-06-10T02:17:00Z [ERROR] Bulk disconnect event: 842 sessions dropped in 60s (threshold 50)
2026-06-10T02:17:10Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_9d2e1f user_id=u_a4b9c8
2026-06-10T02:17:20Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_c8f2b1 user_id=u_d1e7f3
2026-06-10T02:17:30Z [ERROR] WebSocket session disconnect: upstream read_states timeout session_id=sess_4a1d9c user_id=u_b2c8e4
2026-06-10T02:17:45Z [ERROR] Client reconnect storm: 4821 sessions reconnecting in 30s window (threshold 500)
2026-06-10T02:17:50Z [ERROR] Reconnect authentication flood: 4821 IDENTIFY payloads queued
2026-06-10T02:18:00Z [WARNING] message-svc event fanout queue depth: 31204 pending (baseline 400)
2026-06-10T02:18:10Z [ERROR] HTTP client pool to read-states-svc under pressure: 180/200 in use
2026-06-10T02:18:30Z [ERROR] HTTP client pool to message-svc under pressure: 190/200 in use
2026-06-10T02:19:00Z [ERROR] Reconnect storm continuing: 3201 sessions reconnecting in 30s window
2026-06-10T02:19:20Z [ERROR] HTTP client pool to message-svc EXHAUSTED (200/200 in use) — new fanout requests queuing
2026-06-10T02:19:35Z [ERROR] Event fanout timeout for session_id=sess_f1a2b3: no message-svc slot available after 2000ms
2026-06-10T02:19:40Z [ERROR] Event fanout timeout for session_id=sess_d4e5f6: no message-svc slot available after 2000ms
2026-06-10T02:20:00Z [ERROR] Connected clients: 71204 (-14797 from peak) — active disconnects ongoing
2026-06-10T02:21:00Z [ERROR] Connected clients: 68021 (-17980 from peak) — read_states upstream still timing out
2026-06-10T02:23:00Z [ERROR] Connected clients: 62441 — read_states upstream still timing out
2026-06-10T02:25:00Z [ERROR] Connected clients: 61208 — read_states upstream still timing out
2026-06-10T02:30:00Z [ERROR] Connected clients: 60112 — sustained degradation
2026-06-10T02:35:00Z [WARNING] Connected clients: 61804 — slight recovery, reconnects stabilising
2026-06-10T02:38:00Z [INFO] read_states mark_read API latency normalized: P99=45ms (upstream recovered)
2026-06-10T02:38:05Z [INFO] mark_read upstream timeout rate: 0.2% (nominal)
2026-06-10T02:38:10Z [INFO] Client reconnect rate dropping: 180 sessions reconnecting in 30s window
2026-06-10T02:38:30Z [INFO] HTTP client pool to read-states-svc recovered: 14/200 in use
2026-06-10T02:39:00Z [INFO] HTTP client pool to message-svc recovered: 8/200 in use
2026-06-10T02:41:00Z [INFO] Reconnect storm subsided: 41 sessions reconnecting in 30s window
2026-06-10T02:43:00Z [INFO] Connected clients: 83201 — recovery in progress
2026-06-10T02:46:00Z [INFO] Health check ok — connected clients: 84892, read_states P99: 12ms
2026-06-10T02:48:00Z [INFO] Connected clients: 85104 — fully recovered
30 changes: 30 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/live/message-svc.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
2026-06-10T02:10:00Z [INFO] MessageService started, version 5.2.8
2026-06-10T02:10:01Z [INFO] Worker thread pool initialized: 512 threads
2026-06-10T02:10:02Z [INFO] Connected to gateway cluster (load balanced, 12 gateway instances)
2026-06-10T02:11:00Z [INFO] Health check ok — inbound rate: 848 req/s, queue depth: 312
2026-06-10T02:12:00Z [INFO] Health check ok — inbound rate: 852 req/s, queue depth: 289
2026-06-10T02:13:00Z [INFO] Health check ok — inbound rate: 861 req/s, queue depth: 301
2026-06-10T02:14:00Z [INFO] Health check ok — inbound rate: 874 req/s, queue depth: 320
2026-06-10T02:17:00Z [INFO] Health check ok — inbound rate: 1240 req/s, queue depth: 4812
2026-06-10T02:18:00Z [INFO] Health check ok — inbound rate: 4480 req/s, queue depth: 24312
2026-06-10T02:18:10Z [WARNING] Inbound connection rate from gateway: 12400 req/s (baseline 850 req/s) — gateway reconnect storm detected
2026-06-10T02:18:20Z [WARNING] Worker thread pool utilization: 448/512 (87%) — approaching saturation
2026-06-10T02:18:35Z [WARNING] Fanout queue depth: 48120 pending events (threshold 10000)
2026-06-10T02:18:50Z [WARNING] Worker thread pool utilization: 501/512 (98%) — critical
2026-06-10T02:19:05Z [ERROR] Worker thread pool saturated: 512/512 threads in use — new requests rejected
2026-06-10T02:19:10Z [ERROR] Message fanout rejected for channel_id=ch_9f2e1b: no worker threads available
2026-06-10T02:19:15Z [ERROR] Message fanout rejected for channel_id=ch_4a8c2d: no worker threads available
2026-06-10T02:19:20Z [ERROR] Message fanout rejected for channel_id=ch_1b7e3f: no worker threads available
2026-06-10T02:19:30Z [ERROR] Message fanout timeout for channel_id=ch_9f2e1c: upstream slow (2000ms deadline exceeded)
2026-06-10T02:19:45Z [ERROR] Message fanout timeout for channel_id=ch_6b3a4e: upstream slow (2000ms deadline exceeded)
2026-06-10T02:20:00Z [ERROR] Worker thread pool saturated: 512/512 threads in use — queue depth 84210
2026-06-10T02:22:00Z [ERROR] Worker thread pool saturated: 512/512 threads in use — queue depth 91044
2026-06-10T02:25:00Z [ERROR] Worker thread pool saturated: 512/512 threads in use — queue depth 88302
2026-06-10T02:30:00Z [WARNING] Worker thread pool utilization: 498/512 — still elevated, reconnect storm persisting
2026-06-10T02:35:00Z [WARNING] Worker thread pool utilization: 420/512 — improving
2026-06-10T02:38:15Z [INFO] Inbound rate from gateway normalizing: 1840 req/s
2026-06-10T02:38:20Z [INFO] Worker thread pool: 248/512 in use — pressure releasing
2026-06-10T02:38:30Z [INFO] Fanout queue draining: 12401 pending
2026-06-10T02:39:00Z [INFO] Health check ok — inbound rate: 1120 req/s, queue depth: 4802
2026-06-10T02:41:00Z [INFO] Health check ok — inbound rate: 882 req/s, queue depth: 388
2026-06-10T02:42:00Z [INFO] Health check ok — inbound rate: 861 req/s, queue depth: 291 — nominal
22 changes: 22 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/live/presence-svc.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
2026-06-10T02:10:00Z [INFO] PresenceService started, version 2.4.1
2026-06-10T02:10:01Z [INFO] Connected to presence store (Redis cluster, 6 shards)
2026-06-10T02:11:00Z [INFO] Health check ok — presence updates: 480/s, Redis P99: 3ms
2026-06-10T02:12:00Z [INFO] Health check ok — presence updates: 492/s, Redis P99: 3ms
2026-06-10T02:13:00Z [INFO] Health check ok — presence updates: 501/s, Redis P99: 4ms
2026-06-10T02:14:00Z [INFO] Health check ok — presence updates: 488/s, Redis P99: 3ms
2026-06-10T02:15:00Z [INFO] Health check ok — presence updates: 495/s, Redis P99: 3ms
2026-06-10T02:17:00Z [WARNING] Presence update flood: 4200 state changes/s (threshold 500/s) — consistent with large client reconnect event
2026-06-10T02:17:10Z [WARNING] Inbound OFFLINE events: 4821/30s (clients disconnecting from gateway)
2026-06-10T02:17:30Z [WARNING] Inbound ONLINE events: 3901/30s (clients reconnecting to gateway)
2026-06-10T02:18:00Z [WARNING] Presence update queue depth: 18240 — processing backlog
2026-06-10T02:18:30Z [WARNING] Redis pipeline latency P99: 42ms (threshold 10ms) — write contention under flood
2026-06-10T02:19:00Z [WARNING] Presence update flood: 3800 state changes/s — sustained
2026-06-10T02:19:22Z [WARNING] Rate-limiting presence updates: dropping 18% of state changes to protect Redis cluster
2026-06-10T02:20:00Z [WARNING] Presence update flood: 3200 state changes/s
2026-06-10T02:22:00Z [WARNING] Presence update flood: 2800 state changes/s
2026-06-10T02:25:00Z [WARNING] Presence update flood: 2400 state changes/s — slowly subsiding
2026-06-10T02:30:00Z [WARNING] Presence update rate: 1200 state changes/s
2026-06-10T02:35:00Z [INFO] Health check ok — presence updates: 820/s, Redis P99: 5ms
2026-06-10T02:38:00Z [INFO] Presence update rate returning to normal: 540/s
2026-06-10T02:41:10Z [INFO] Health check ok — presence updates: 498/s, Redis P99: 3ms — nominal
2026-06-10T02:42:00Z [INFO] Health check ok — presence updates: 491/s, Redis P99: 3ms
36 changes: 36 additions & 0 deletions eval/dataset_4_discord_gateway_cascade/live/read-states-svc.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
2026-06-10T02:10:00Z [INFO] Service started, version 3.8.1 (Cassandra: 4.1.3)
2026-06-10T02:10:01Z [INFO] Connected to Cassandra cluster (nodes: 10.0.1.6, 10.0.1.7, 10.0.1.8)
2026-06-10T02:10:02Z [INFO] Keyspace read_states: replication_factor=3, consistency=QUORUM
2026-06-10T02:11:00Z [INFO] Health check ok — batch write latency P99: 12ms
2026-06-10T02:12:00Z [INFO] Health check ok — batch write latency P99: 14ms
2026-06-10T02:13:00Z [INFO] Health check ok — batch write latency P99: 11ms
2026-06-10T02:14:00Z [INFO] Processing 142 read-state updates (user mark-reads, last-viewed)
2026-06-10T02:14:30Z [INFO] Health check ok — batch write latency P99: 18ms
2026-06-10T02:14:48Z [WARNING] Cassandra BATCH APPLY latency spike: 2847ms (threshold 500ms), coordinator=10.0.1.8
2026-06-10T02:14:52Z [WARNING] Cassandra BATCH APPLY latency spike: 4103ms (threshold 500ms), coordinator=10.0.1.8
2026-06-10T02:15:00Z [ERROR] Cassandra timeout on BATCH APPLY to read_states keyspace: OperationTimedOutException (coordinator=10.0.1.8, consistency=QUORUM, received=1, required=2)
2026-06-10T02:15:01Z [ERROR] Failed to persist read-state for user_id=u_9f2a1c, channel_id=ch_8821: write timeout
2026-06-10T02:15:02Z [ERROR] Cassandra timeout on BATCH APPLY to read_states keyspace: OperationTimedOutException (coordinator=10.0.1.8, consistency=QUORUM, received=1, required=2)
2026-06-10T02:15:03Z [ERROR] Failed to persist read-state for user_id=u_c3b8e2, channel_id=ch_4491: write timeout
2026-06-10T02:15:05Z [WARNING] Cassandra node 10.0.1.8 not responding to gossip — possible partition leader election in progress
2026-06-10T02:15:12Z [ERROR] Cassandra timeout on BATCH APPLY to read_states keyspace: OperationTimedOutException (coordinator=10.0.1.8, consistency=QUORUM, received=1, required=2)
2026-06-10T02:15:30Z [ERROR] Cassandra connection pool to coordinator 10.0.1.8 degraded: 28/50 connections healthy
2026-06-10T02:16:00Z [ERROR] Cassandra timeout on BATCH APPLY: coordinator=10.0.1.8 unreachable, retrying with 10.0.1.7
2026-06-10T02:16:15Z [ERROR] All coordinators timeout: QUORUM write requires 2 nodes but only 1 acknowledged (coordinator=10.0.1.7)
2026-06-10T02:17:00Z [ERROR] Cassandra write queue depth: 8421 pending BATCH APPLY operations
2026-06-10T02:17:30Z [ERROR] Cassandra write queue depth: 18204 pending BATCH APPLY operations — queue growing faster than drain rate
2026-06-10T02:18:00Z [ERROR] read_states write queue overflow: dropping oldest 5000 pending writes to prevent OOM
2026-06-10T02:20:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:22:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:25:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:28:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:30:00Z [WARNING] Cassandra node 10.0.1.8 gossip re-established — partition leader election in progress
2026-06-10T02:33:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:36:00Z [ERROR] Cassandra timeout on BATCH APPLY: OperationTimedOutException (received=1, required=2)
2026-06-10T02:38:30Z [INFO] Cassandra coordinator re-elected, new partition leader: 10.0.1.9
2026-06-10T02:38:35Z [INFO] Cassandra QUORUM writes resumed — batch write latency P99: 28ms
2026-06-10T02:38:40Z [INFO] Processing 41821 queued read-state updates (backlog drain started)
2026-06-10T02:39:00Z [INFO] Health check ok — batch write latency P99: 22ms
2026-06-10T02:39:30Z [INFO] Backlog drain progress: 12400/41821 updates processed
2026-06-10T02:41:00Z [INFO] Backlog drain complete — all pending read-state updates written
2026-06-10T02:42:00Z [INFO] Health check ok — batch write latency P99: 13ms
Loading
Loading