Skip to content

Stale per-repo issue-discovery fields survive a successful scoring round and inflate roll-up totals #1610

Description

@kevin8600

Description

run_issue_discovery has two per-round reset paths for a miner's issue-discovery data, and they diverge:

  • The no-issues path (_clear_issue_discovery_fields, scan.py) zeroes the issue fields on every repo_evaluation.
  • The scoring path (_finalize_repo_issue_scores) only rewrites the repos seen this round (sorted(set(repo_acc) | set(open_counts))), then _roll_up_issue_totals sums over all repo_evaluations.

As a result, a repo that was issue-scored in a prior round but is absent this round keeps its stale per-repo issue values, and those get summed back into the round-level totals.

This is reachable because an OSS-fetch-failed miner is restored from the evaluation cache (store_or_use_cached_evaluation in neurons/validator.py replaces the eval with the cached one, which carries prior-round repo_evaluations). Issue discovery then runs on that restored eval — and forward.py explicitly expects it to produce fresh issue-discovery fields — but the scoring path leaves the now-inactive repo stale.

Steps to Reproduce

  1. Round N-1: a miner (with ≥1 merged PR, so it's cacheable) is issue-scored in repo A → the cache stores repo_evaluations['A'] with non-zero total_solved_issues / issue_discovery_score / is_issue_eligible.
  2. Round N: the miner's OSS PR fetch fails (github_pr_fetch_failed), so the evaluation is restored from cache (carrying repo A's stale issue fields).
  3. Round N: the issue-mirror fetch succeeds, but the miner now only has issues in a different repo B (and no open issues in A), so A is absent from both repo_acc and open_counts.
  4. _finalize_repo_issue_scores rewrites only B; repo A is never touched; _roll_up_issue_totals sums A (stale) + B (fresh).

Expected Behavior

The scoring path zeroes stale per-repo issue fields for repos not seen this round (same as the no-issues path), so the round-level roll-up reflects only this round's issues.

Actual Behavior

Repo A retains total_solved_issues=7, issue_discovery_score=8.12, is_issue_eligible=True; the round-level total_solved_issues / issue_discovery_score / is_issue_eligible inherit those stale values. The corrupted per-repo rows are persisted via bulk_store_evaluation and re-written to the evaluation cache via update_issue_discovery, so the error compounds across rounds. (Current-round emission allocation reads the freshly-rebuilt issue_discovery_issues list, so this is a stored-data / reporting correctness bug — CLI miner score, analytics, and the cross-round cache — rather than an emissions bug.)

Environment

  • OS: Linux (CI) / cross-platform
  • Python version: 3.12
  • Commit/Version: 822056d (v5.0.0)

Additional Context

The no-issues path already does the correct full reset (_clear_issue_discovery_fields loops over every repo_eval), which shows the intended contract; the scoring path simply drifted from it. Fix: extract the per-repo reset into a shared helper and apply it on both paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions