Skip to content

feat(packaging): add DOMjudge problem packager#598

Open
rsalesc wants to merge 2 commits into
mainfrom
worktree-domjudge-packager
Open

feat(packaging): add DOMjudge problem packager#598
rsalesc wants to merge 2 commits into
mainfrom
worktree-domjudge-packager

Conversation

@rsalesc

@rsalesc rsalesc commented Jun 12, 2026

Copy link
Copy Markdown
Owner

Summary

Adds rbx package domjudge, building a DOMjudge-importable problem zip. The layout follows the ICPC problem package format plus DOMjudge extensions, mirroring what pol2dom produces (known to import cleanly):

domjudge-problem.ini      # short-name, name, exact fractional timelimit, contest color
problem.yaml              # limits {memory, output}, validation[, validator_flags]
problem.pdf               # the already-built main statement PDF
data/sample/, data/secret # 001.in/001.ans per-directory counters
output_validators/        # custom checker (flattened) + patched testlib.h + rbx.h
submissions/              # jury solutions by expected outcome

Design doc: docs/plans/2026-06-12-domjudge-packager-design.md.

Key decisions

  • Checker mapping: checkers resolving to the bundled builtins map to DOMjudge's default output validator (wcmp/ncmp/yesno → no flags, dcmpfloat_tolerance 1e-6). Anything else — including a same-named local file, which may have been user-edited — ships as a custom validator, flattened via flattening.build_flat_namespace (cross-dir includes rewritten).
  • testlib patch (domjudge/testlib_patch.py): DOMjudge validators speak the Kattis protocol (exit 42/43, team output on stdin, feedback dir), which vanilla testlib doesn't. Ports pol2dom's patch (from cn-xcpc-tools/testlib-for-domjudge), applied to the bundled testlib at package time; raises if an anchor is missing so a testlib upgrade fails tests, not packages. Verified locally: patched wcmp compiles and exits 42/43 with judgemessage.txt under the DOMjudge calling convention.
  • Solutions: ACCEPTED→accepted, WA→wrong_answer, TLE→time_limit_exceeded, RTE/MLE→run_time_error (DOMjudge reports MLE as RTE); ambiguous outcomes skipped with a console note.
  • Limits: uses the domjudge limits profile when saved, else package limits (not required, unlike BOCA — DOMjudge has a single per-problem TL).
  • Scope: BATCH only; rbx COMMUNICATION pairs interactor+checker, which doesn't map onto DOMjudge's single output validator (follow-up). Build only — no upload.

Testing

  • 12 new tests in tests/rbx/box/packaging/test_domjudge.py (testlib patch against bundled testlib, ini/yaml content, builtin-vs-custom checker resolution, flattening, submissions mapping, sample/secret routing, full package() smoke).
  • tests/rbx/box/packaging + tests/rbx/box/completion: 1139 passed (3 docker-compose BOCA e2e errors are pre-existing local-env failures).
  • Completion spec regenerated (mise run gen-completion-spec); drift test green.
  • Docs: new docs/setters/packaging/domjudge.md + index table row + mkdocs nav.

🤖 Generated with Claude Code

`rbx package domjudge` builds a DOMjudge-importable zip following the
ICPC problem package format plus DOMjudge extensions, mirroring the
layout pol2dom produces: domjudge-problem.ini + problem.yaml metadata,
the already-built statement PDF as problem.pdf, data/sample + data/secret
testcases, jury solutions under submissions/, and custom checkers
flattened into output_validators/ with a testlib.h patched to speak the
Kattis validator protocol (exit 42/43, team output on stdin, feedback
dir). Builtin wcmp/ncmp/yesno/dcmp checkers map to DOMjudge's default
validator instead. BATCH problems only for now.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'yesno.cpp': None,
'dcmp.cpp': 'float_tolerance 1e-6',
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please let's not resort to domjudge default output validators. let's always honor the rbx checker.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 087f800. Dropped the default-validator mapping entirely — the rbx checker is now always shipped as a custom output validator (validation: custom), so DOMjudge judges with exactly the checker rbx uses locally. This applies to the builtins (wcmp, ncmp, …) too. Verified locally that a builtin checker compiles and exits 42/43 under the DOMjudge calling convention after the testlib patch.

Comment thread rbx/box/packaging/domjudge/packager.py Outdated
ExpectedOutcome.MEMORY_LIMIT_EXCEEDED: 'run_time_error',
}

_CPP_SUFFIXES = {'.cpp', '.cc', '.cxx', '.c++'}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use the language kinds for the checker to identify this

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 087f800. Now uses LanguageKind.CPP in environment.language_kinds(code.find_language(checker)) instead of the hardcoded suffix set, so detection derives from the actual toolchain and is robust to custom language names. Added test_output_validators_reject_non_cpp_checker to cover the rejection path.

Comment thread rbx/box/packaging/domjudge/packager.py Outdated
}

# DOMjudge reports MLE as RTE by default, hence the MLE mapping.
_SUBMISSION_DIRS = {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you explain this better? i'm not sure i like the idea of solutions with certain outcomes just vanishing from the upload. please research how these submissions are used, when they matter, which ones we can map to their outcomes, etc. please specifically validate that domjudge (not only problem package format) supports what you're stating.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right — dropping solutions was wrong. I researched how DOMjudge actually handles jury submissions (verified against the DOMjudge 9.0 source, not just the problem-package spec) and reworked it in 087f800 so no solution is ever dropped.

How DOMjudge uses these: on import it auto-judges every solution in submissions/ and records an expected verdict per submission, then surfaces any mismatch on the jury Judging verifier page (it never blocks the import). The expected verdict comes from one of two places:

  • Directory name, when it normalizes to a real verdict via DOMjudge's PROBLEM_RESULT_REMAP table (SubmissionService.php). That's 7 dirs, not 4: accepted, wrong_answer, time_limit_exceeded, run_time_error, output_limit, compiler_error, no_output.
  • An @EXPECTED_RESULTS@: source annotation, which supports multiple acceptable verdicts (e.g. CORRECT, TIMELIMIT). Crucially, in a verdict-named dir DOMjudge collapses the annotation to the dir's single verdict; in a non-verdict dir (e.g. mixed/) it keeps the full list. So multi-verdict expectations must live in a non-verdict dir.

New mapping (every outcome shipped):

rbx outcome placement
ACCEPTED / WRONG_ANSWER / TIME_LIMIT_EXCEEDED / RUNTIME_ERROR matching standard dir, no annotation
OUTPUT_LIMIT_EXCEEDED output_limit/ (DOMjudge extension dir)
ACCEPTED_OR_TLE mixed/ + @EXPECTED_RESULTS@: CORRECT, TIMELIMIT
TLE_OR_RTE mixed/ + TIMELIMIT, RUN-ERROR
INCORRECT mixed/ + all non-CORRECT verdicts
ANY mixed/ + all verdicts
MEMORY_LIMIT_EXCEEDED mixed/ + RUN-ERROR, TIMELIMIT

The one thing DOMjudge genuinely can't represent: there is no memory-limit verdict and no @EXPECTED_RESULTS@ token for it — an over-memory run surfaces as RTE (sometimes TLE). So MLE is the single lossy mapping; I encode it as RUN-ERROR, TIMELIMIT and documented the caveat. Everything else is exact.

Annotation comment prefix is # for Python, // otherwise. Docs/CLAUDE.md updated with the table and a note that mixed/ solutions trigger a harmless "result does not match directory" message on import. Covered by test_submissions_single_verdict_use_standard_dirs and test_submissions_ambiguous_outcomes_use_mixed_dir_with_annotation.

- Always ship the rbx checker as a custom output validator
  (validation: custom); never fall back to DOMjudge's default
  validators, so DOMjudge judges with the same checker rbx uses
  locally. Drops the wcmp/ncmp/yesno/dcmp -> default mapping.
- Detect the C++ checker via language kinds
  (LanguageKind.CPP in environment.language_kinds(find_language))
  instead of hardcoded file suffixes.
- Ship every solution; none are dropped. Single-verdict outcomes go
  to the matching standard submission directory (incl. output_limit
  for OLE); multi-verdict outcomes (MLE, ACCEPTED_OR_TLE, TLE_OR_RTE,
  INCORRECT, ANY) go to submissions/mixed/ with an @EXPECTED_RESULTS@
  annotation listing every acceptable DOMjudge verdict. MLE is the
  one lossy mapping (DOMjudge has no memory-limit verdict).

Updates docs, CLAUDE.md and the design doc to match the corrected
DOMjudge submission semantics (directory name -> verdict via remap
table; annotation honored verbatim in non-verdict dirs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant