fix(distributed): Harden evaluator claim recovery and warmup gating by fuyu0425 · Pull Request #208 · sslab-gatech/CRSBench

fuyu0425 · 2026-04-25T12:11:23Z

Summary

Synchronize evaluator claim reservations and active-claim tracking so warmup only feeds durable claimed work.
Re-enqueue or refresh terminal verify jobs, clean up canceled verify-job residue, and fail reclaim when stale opaque wrappers remain.
Tighten distributed regression coverage across warmup, claim worker, queue cleanup, and run-experiment paths.

Testing

uv run pytest -q tests/test_evaluator_claim_worker.py tests/test_queue_cleanup.py tests/test_run_experiment_distributed.py
scripts/ci-tests/run-local.sh checks

Notes

Includes the fix/evaluator-warning branch plus a follow-up fix for stale verify-wrapper reclaim behavior on opaque queues.

fuyu0425 added 22 commits April 25, 2026 05:34

fix(monitor): harden rich pager tty input

aaeadc8

fix(monitor): preserve rich pager hotkeys on fd failures

21f74ca

fix(monitor): handle concurrent rich pager inputs

9f68855

test(monitor): cover nonblocking rich pager reads

f036fa6

test(monitor): relax nonblocking read assertions

9c6c1eb

test(monitor): verify nonblocking read effects

9519180

test(monitor): cover rich input state recovery

c5338c6

test(monitor): cover blockingioerror continuation

09bca89

fix(monitor): keep rich paging responsive during refresh

8694e0f

fix(monitor): pause and pace rich auto-rotation

ed6702e

fix(monitor): bound rich poller shutdown

fb7c6d8

fix(monitor): mark paused rich paging state

82f7ac2

fix(monitor): stabilize rich paused paging footer

4a73ec1

test(monitor): assert rendered rich footer output

42b8011

fix(monitor): preserve footer help on narrow terminals

95834d3

fix(monitor): keep rich footer readable on short terminals

a388e75

fix(cloud): exclude staged trial copies from collect

c154b83

fix(cloud): keep staged content out of collect

390f5f4

fix(cloud): scope staged symlink filtering to trial root

95a5b68

fix(cloud): seal staged artifact reentry paths

2793547

fix(cloud): harden staged symlink filtering

6df0981

fix(cloud): tighten report-log symlink handling

8a7903a

fuyu0425 force-pushed the fix/evalautor-error branch from 77ada6c to bf51c56 Compare April 25, 2026 12:30

fuyu0425 added 7 commits April 25, 2026 08:35

fix(cloud): scope staged collect filtering to real trial roots

da39053

fix(cloud): validate collect trial roots against layout

15e0ed5

fix(cloud): reject pseudo trial roots in collect

1feb823

fix(cloud): require metadata-backed collect trial roots

b37dc65

fix(cloud): backstop collect trial roots with corrupt metadata

d38678f

fix(cloud): align report-log restore allowlist

a505eb5

fix(cloud): scope corrupt-metadata trial fallback by prefix

183aa02

fuyu0425 added 25 commits April 25, 2026 09:15

fix(cloud): require exact trial namespace for fallback

65005fa

fix(cloud): validate fallback against repo trial layouts

df708a7

fix(cloud): trust published trial roots during local recollect

fcb013b

fix(cloud): require local corrupt-trial shape for fallback

68ecb06

test(cloud): cover fallback marker edge cases

fbeeb33

test(cloud): cover direct published-root fallback cases

58bc7ae

fix(distributed): synchronize evaluator active claims

dce5e8d

fix(distributed): reserve evaluator claim slots

f090adb

fix(distributed): track claim reservations per attempt

d8ddfb4

fix(distributed): gate warmup on claimed materialization

ada3b4f

fix(distributed): gate warmup on durable claim state

81c4a4a

fix(distributed): recheck warmup enqueue under claim gate

2992720

fix(distributed): unblock warmup on stale claims and duplicates

bd33d11

test(distributed): cover stale and duplicate warmup gating

d998259

fix(distributed): reenqueue terminal claimed verify jobs

bb937d3

test(distributed): tighten claimed verify reclaim coverage

db742f6

fix(distributed): clear canceled verify job residue

c68ff93

fix(distributed): discover canceled jobs during cleanup

c98394f

fix(distributed): keep canceled jobs out of logical resume state

9db6015

fix(distributed): ignore canceled residue in continue mode

415d063

test(distributed): cover canceled queue-mode boundaries

deff15c

test(distributed): tighten canceled residue queue-mode coverage

06f4526

fix(distributed): fail reclaim on stale verify wrappers

2fe71f2

test(distributed): cover opaque verify wrapper refresh

bf74a8b

test(distributed): cover opaque wrapper enqueue refresh

2456a20

fuyu0425 force-pushed the fix/evalautor-error branch from bf51c56 to 2456a20 Compare April 25, 2026 13:45

fuyu0425 merged commit 8e23f18 into main Apr 25, 2026
5 checks passed

fuyu0425 deleted the fix/evalautor-error branch April 25, 2026 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(distributed): Harden evaluator claim recovery and warmup gating#208

fix(distributed): Harden evaluator claim recovery and warmup gating#208
fuyu0425 merged 54 commits into
mainfrom
fix/evalautor-error

fuyu0425 commented Apr 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fuyu0425 commented Apr 25, 2026

Summary

Testing

Notes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant