Add Martian benchmark results and public positioning#48
Merged
Conversation
Architecture changes validated on Martian Code-Review-Bench with GLM-5.2: - Decomposed consistency-verify: extract cross-location consistency obligations, then one focused harness verifier per obligation. Emergent depth lever for cross-file/structural bug families (no per-problem prompts). - Evidence-pack-first reviewers: pre-extract target+imports+callers once and inject, flipping "read the repo" to "investigate this pack, escalate if needed" (BudgetConfig.evidence_pack_reviewers, file-cached by mtime). - Post-worthiness gate: optional recall-preserving pre-layer gate (CommentConfig.post_worthiness_gate) returning keep_indices. - Coverage and consistency phases run concurrently via asyncio.gather; file-patch build memoized per run. - fetch_pr retry loop (4 attempts; TransportError/5xx/403/429) with follow_redirects so cal.com 301s and transient Sentry errors don't abort.
PR-AF run end-to-end on a single open model (GLM-5.2) against the 50-PR Martian Code-Review-Bench. On the 38 runnable PRs: #1 in valid findings delivered (~3x the leading commercial tools), #2 of 42 in golden recall (0.706), co-leader on honest F1 (0.82). - README.md / RESULTS.md -- guide + headline results - results/<id>.json (38) -- per-problem exact, untrimmed findings + per-golden judge verdicts - scoreboard.{md,jsonl}, problems.json -- aggregate recall + ranked worklist - analysis/ -- leaderboard standing + substantive scoreboard - scripts/ -- node launcher, campaign, ensemble, scoring gitignore the whole _glm52_bench/ scratch dir (raw transcripts, logs, per-run caches); canonical reproduction scripts live committed under scripts/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch packages the Martian Code-Review-Bench work and updates the public README so PR-AF is presented around its strongest benchmark position: #1 open-source code reviewer on Code-Review-Bench with GLM-5.2, plus model-flexible/self-hosted usage.
What changed
benchmark/martian-code-review-bench/with headline docs, per-PR result JSON, machine-readable scoreboards, analysis views, and reproduction scripts..DS_Storeis deleted and ignored; draft hero assets and benchmark scratch/cache folders are not tracked.Public benchmark organization
README.md: overview and methodology for the benchmark package.RESULTS.md: headline results and adjusted scoring view.scoreboard.md/scoreboard.jsonl: human-readable and machine-readable aggregate results.problems.json: ranked 50-PR worklist.results/*.json: per-PR PR-AF findings and judge verdicts.analysis/: secondary leaderboard and substantive-golden views.scripts/: reproduction, campaign, ensemble, and scoring scripts.Validation
git diff --check.venv/bin/pytest-> 69 passed