fix(judge): exclude .txt files from the LLM judge corpus

[sanchitmehtagit]

Summary

Exclude .txt files from the corpus sent to the LLM judge, matching the existing .md exclusion.

Agents routinely emit large standalone .txt docs (implementation summaries, verification checklists) alongside the real source. These aren't code, add no signal the judge needs, and inflate the judged corpus — sometimes past judge.maxCodeChars, which errors the judge grader out and scores it as a hard fail.

Concrete failure this fixes

In the vue_quickstart run, the judged corpus was 44,931 chars — over the 32,768 limit. Both LLM-judge graders errored with Code corpus exceeds limit and the run flipped to status=failure despite 18/20 deterministic graders passing. The overflow was driven by agent-generated docs — a ~13KB IMPLEMENTATION_SUMMARY.txt and a VERIFICATION_CHECKLIST.txt. Stripping .txt (as .md already is) drops the corpus under the limit so the judge actually runs.

This is happening with only Haiku 4.5 model

Change

• Add /\.txt$/i to JUDGE_EXCLUDED_PATTERNS in llm-judge.ts, with a comment explaining the rationale.
• Add a unit test covering .txt exclusion.

Safety

The judge filter (isJudgeExcluded) is applied only inside the LLM-judge executor. Deterministic contains/notContains graders read the full file set (collectFiles), so they are unaffected by this change.

Test

npx turbo run test --filter=@a0/evals-core — 486 passed (25 files), including the new .txt exclusion test.

Summary by CodeRabbit

• Bug Fixes
  • Improved evaluation accuracy by excluding .txt files from LLM-judge inputs, alongside other non-relevant project files.
  • Added support for excluding uppercase .TXT files and common text-based summary or notes files.

[coderabbitai]

📝 Walkthrough

Walkthrough

The LLM judge now excludes .txt files from its input corpus, documents the updated exclusion behavior, and tests representative uppercase text filenames.

Changes

LLM judge input exclusions

Layer / File(s)	Summary
Text file exclusion rule and coverage packages/evals-core/src/graders/executors/llm-judge.ts, packages/evals-core/tests/graders/executors.test.ts	Adds a case-insensitive .txt exclusion pattern, expands the related documentation, and verifies exclusion of representative text document filenames.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: frederikprijck

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: excluding .txt files from the LLM judge corpus.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch flywheel/judge-exclude-txt-files

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}