Thanks for working on Bencher!
The new GitHub checks feature has been really useful, but unfortunately it only updates the check status once it completes. has a few issues:
- When the test job reruns (user pushes to the PR branch), a failure/success status lingers until the run is complete, giving false reading in the meantime.
- If the test job was cancelled/stopped for any reason, the old failure/success is left reported on the PR.
- This issue worsens for long-running/slow benchmarks (like callgrind/gangraun) that can take up to 1~2 hours to finish.
I suggest setting the GitHub check status to in_progress before running the benchmark, so that the status is reflected accordingly until results are ready.
Thanks for working on Bencher!
The new GitHub checks feature has been really useful, but unfortunately it only updates the check status once it completes. has a few issues:
I suggest setting the GitHub check status to
in_progressbefore running the benchmark, so that the status is reflected accordingly until results are ready.