kata-benchkit

kata-benchkit is the maintainer tool for building and rotating Kata benchmark tasks.

It does not run miner competitions. It helps maintainers create the public and private task pools that Kata later uses.

What It Is For

kata-benchkit helps with:

• mining candidate tasks from real repo history
• scaffolding Kata-compatible task folders
• linting tasks for production use
• checking pack-level readiness
• generating the first public/private pool
• rotating the hidden private pool later
• exporting fully retired private tasks into the public side

Repo Split It Assumes

• public kata-benchmarks
  • public tasks only
• private kata-benchmarks-private
  • hidden holdout tasks only

The current king agent code is not handled here. It lives in the kata repo.

CLI Commands

Top-level command groups:

• registry
  • bootstrap a benchmark registry root
• task
  • create, lint, retire, or export one task
• pack
  • report pack readiness and set pack targets
• history
  • scan a real repo history for candidate tasks
• pool
  • generate or rotate the public/private live pools

Useful commands:

• kata_benchkit registry init
• kata_benchkit pool generate-initial
• kata_benchkit pool rotate
• kata_benchkit pool report
• kata_benchkit task init
• kata_benchkit task lint
• kata_benchkit task retire
• kata_benchkit task export-public
• kata_benchkit pack report
• kata_benchkit pack targets set
• kata_benchkit history scan

Current Competition Design

For the current live design:

• public pool size: 20
• private holdout pool size: 10
• each duel uses 30 tasks total

The public side grows over time. During a duel, Kata selects 20 tasks randomly from the live public pool.

The private side stays hidden. Maintainers rotate it manually when they want a new hidden pool.

Quickstart

Bootstrap both registries:

uv run python -m kata_benchkit registry init --root /path/to/kata-benchmarks
uv run python -m kata_benchkit registry init --root /path/to/kata-benchmarks-private

Generate the first 20 public + 10 private pool:

uv run python -m kata_benchkit pool generate-initial \
  --repo /path/to/taopedia-articles \
  --repo-pack e35ventura__taopedia-articles \
  --public-root /path/to/kata-benchmarks \
  --private-root /path/to/kata-benchmarks-private \
  --history-limit 200 \
  --check-command "npm run format:check" \
  --check-command "npm run validate"

pool generate-initial defaults to 20 public tasks and 10 private holdout tasks. Pass --public-count or --private-count only when testing a smaller local fixture.

Rotate the hidden pool later:

uv run python -m kata_benchkit pool rotate \
  --repo /path/to/taopedia-articles \
  --repo-pack e35ventura__taopedia-articles \
  --public-root /path/to/kata-benchmarks \
  --private-root /path/to/kata-benchmarks-private \
  --history-limit 200 \
  --check-command "npm run format:check" \
  --check-command "npm run validate"

pool rotate defaults to generating 10 new private holdout tasks. During rotation, the previous hidden pool is revealed into the public pack as retired tasks, and the fresh private pool is selected without reusing existing public or previous hidden task sources/scopes.

pool rotate is the command that does both:

1. move the old hidden pool toward the public side
2. generate a fresh hidden private pool

It should avoid duplicating tasks that are already public or already revealed.

Report the current pool state:

uv run python -m kata_benchkit pool report \
  --repo-pack e35ventura__taopedia-articles \
  --public-root /path/to/kata-benchmarks \
  --private-root /path/to/kata-benchmarks-private

Task Commands

Create a public task:

uv run python -m kata_benchkit task init \
  --root /path/to/kata-benchmarks \
  --repo-pack e35ventura__taopedia-articles \
  --task-id fix-broken-article-frontmatter \
  --repo-ref https://github.com/e35ventura/taopedia-articles.git@test \
  --source-kind pr \
  --source-ref https://github.com/e35ventura/taopedia-articles/pull/123 \
  --visibility visible \
  --title "Fix article frontmatter and keep validation green" \
  --context "The target article has malformed front matter in the pinned repository snapshot." \
  --goal "Repair the target article metadata so Taopedia validation accepts the page without changing unrelated content." \
  --constraint "Only edit article files under content/pages/articles/." \
  --constraint "Follow the existing Taopedia front matter field names and formatting style." \
  --non-goal "Do not rewrite unrelated article prose or navigation files." \
  --expected-outcome "The target article has valid front matter and the repository validation commands pass." \
  --acceptance-criterion "The target article metadata is valid for the current Taopedia schema." \
  --acceptance-criterion "Unrelated files remain unchanged." \
  --allowed-path "content/pages/articles/" \
  --forbidden-path "content/index/" \
  --check-command "npm run format:check" \
  --check-command "npm run validate"

Create a hidden private task:

uv run python -m kata_benchkit task init \
  --root /path/to/kata-benchmarks-private \
  --repo-pack e35ventura__taopedia-articles \
  --task-id hidden-example \
  --repo-ref https://github.com/e35ventura/taopedia-articles.git@test \
  --source-kind commit \
  --source-ref https://github.com/e35ventura/taopedia-articles/commit/abc123 \
  --visibility holdout \
  --title "Hidden holdout example" \
  --context "The target article contains one repo-specific content defect in the pinned snapshot." \
  --goal "Correct the target article so it matches the intended Taopedia behavior and keeps validation green." \
  --constraint "Only edit content/pages/example/index.mdx unless validation directly requires a paired metadata file." \
  --constraint "Preserve existing front matter structure and citation style." \
  --non-goal "Do not make broad wording cleanup outside the requested defect." \
  --expected-outcome "The intended article defect is fixed without out-of-scope changes." \
  --acceptance-criterion "The corrected article addresses the requested defect." \
  --acceptance-criterion "Repo format and validation checks pass." \
  --allowed-path "content/pages/example/index.mdx" \
  --check-command "npm run format:check" \
  --check-command "npm run validate"

Add deterministic oracle assertions when generic repo checks are not enough:

uv run python -m kata_benchkit task init \
  --root /path/to/kata-benchmarks \
  --repo-pack owner__repo \
  --task-id fix-specific-behavior \
  --repo-ref https://github.com/owner/repo.git@parent-ref \
  --source-kind commit \
  --source-ref https://github.com/owner/repo/commit/fix-ref \
  --visibility visible \
  --title "Fix specific behavior" \
  --context "The parser in src/target.py mishandles one documented edge case in the pinned snapshot." \
  --goal "Update src/target.py so the parser handles the edge case without regressing existing behavior." \
  --constraint "Keep the change limited to src/target.py and directly related tests." \
  --constraint "Follow the existing parser error-handling style." \
  --non-goal "Do not refactor the parser or change unrelated parsing behavior." \
  --expected-outcome "The required behavior is present, the wrong behavior is absent, and tests pass." \
  --acceptance-criterion "The edge case is handled in src/target.py." \
  --acceptance-criterion "Existing parser behavior remains compatible." \
  --allowed-path src/target.py \
  --check-command "pytest" \
  --oracle-target-file src/target.py \
  --oracle-required-contains "src/target.py::fixed behavior" \
  --oracle-forbidden-regex "src/target.py::old_wrong_behavior"

When oracle assertions are supplied, benchkit writes oracle.json and appends python -m kata.oracle verify ... to checks.sh.

Lint one task:

uv run python -m kata_benchkit task lint \
  --path /path/to/kata-benchmarks/benchmarks/e35ventura__taopedia-articles/fix-broken-article-frontmatter

Retirement And Public Export

Private tasks do not become public after one duel.

They should move public only after:

• a newer hidden pool has already taken over
• no live duel can still reference the old hidden pool

Use the lifecycle commands:

uv run python -m kata_benchkit task retire \
  --path /path/to/task \
  --status retired-waiting

uv run python -m kata_benchkit task retire \
  --path /path/to/task \
  --status retired

uv run python -m kata_benchkit task export-public \
  --path /path/to/task \
  --public-root /path/to/kata

Meaning:

• retired-waiting
  • a newer private pool is already active, but old duels may still refer to this task
• retired
  • no live duel should still use this task
• export-public
  • copy the fully retired task into the public archive

Pack And History Commands

Report one pack:

uv run python -m kata_benchkit pack report \
  --path e35ventura__taopedia-articles \
  --root /path/to/kata-benchmarks \
  --private-root /path/to/kata-benchmarks-private

Set pack targets:

uv run python -m kata_benchkit pack targets set \
  --path /path/to/kata-benchmarks/benchmarks/e35ventura__taopedia-articles \
  --visible 20 \
  --holdout 10 \
  --similarity-threshold 0.90 \
  --max-scope-concentration 0.75

Scan history:

uv run python -m kata_benchkit history scan \
  --repo /path/to/target-repo \
  --limit 30

Production Rules

kata-benchkit treats these as important:

• task files exist
• scaffold placeholders are removed
• repo_ref.txt is pinned to a ref with @
• checks.sh is executable and uses set -euo pipefail
• task.md clearly states context, goal, constraints, non-goals, expected outcome, and acceptance criteria
• task.md does not expose hidden oracle/checker internals
• provenance metadata exists in benchkit.json
• visibility and lifecycle status are recorded
• path scope is explicit where possible
• task-specific oracle assertions are present when generic repo checks are too weak
• pack-level visible/holdout targets are enforced
• near-duplicate or over-concentrated task sets are flagged