Add a setting to stop stderr output from failing a testcase#178
Open
znzryb wants to merge 1 commit into
Open
Conversation
Anything a solution wrote to stderr was turned into a RUNTIME ERROR verdict regardless of the exit code, so a single debug print made every testcase fail even when the answer on stdout was correct. Capture the error stream into the verdict instead of aborting on it when the new 'Ignore output written to stderr' setting is on (default), so the verdict depends only on the exit code and stdout, like an online judge does. The captured stderr is printed under the testcase output so debug logs stay visible, and building a solution keeps its previous behaviour.
znzryb
force-pushed
the
ignore-stderr-option
branch
from
July 25, 2026 08:59
60e0774 to
9535b16
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Anything a solution writes to stderr fails the testcase with
[-] FAILURE: RUNTIME ERROR, even when the process exits with0and the answer on stdout is correct.ProcessRunnerthrowsRuntimeErras soon as the error stream is non-empty, so a single debug print makes every testcase — normal run and stress test alike — come back red.Printing intermediate state to stderr is a pretty common debugging habit while stress testing (it keeps the debug log out of the answer that gets compared), and it is exactly what online judges ignore — they verdict a submission on its exit code and stdout only. Right now that habit is unusable: the stress test stops at testcase 1 with a runtime error that says nothing about correctness.
Change
Adds Settings > Tools > AutoCp > Testing > "Ignore output written to stderr", on by default.
When it is on, the error stream is captured into the verdict instead of aborting the run, so a testcase is judged by its exit code and stdout like a judge would do. A non-zero exit code still produces a runtime error, and the captured stderr is now appended to its message, which makes crashes easier to diagnose than before. Turning the setting off restores the previous behaviour of failing any testcase that writes to stderr.
Here the solution dies on an AddressSanitizer heap-buffer-overflow with the setting on: the non-zero exit code (134) still fails the testcase as a runtime error, with the sanitizer report from stderr attached to the message.
Captured stderr is reported through
testStdErrunder the testcase output behind a___[ stderr ]___header, for correct and wrong answers alike. Going through the error channel keeps the debug log in its own color in the test runner, so it stays visually distinct from the answer it sits next to.The same stress test now runs to completion: the testcases whose answer matches pass with their debug log intact, and the one that is genuinely wrong is reported as a wrong answer instead of hiding behind a runtime error.
Compiling a solution is untouched:
TwoStepProcessFactorykeeps the defaultignoreStderr = false, so a failing build still surfaces asBuildErrthrough the same path as before.Notes
participant,judge, generator and correct program), so it covers normal testing, stress testing and interactive problems.ProcessRunner.CapturedResultsgained astderrfield with a default value, and the two verdicts gained a trailingstderrparameter with a default, so existing construction sites keep compiling.