Skip to content

Fix: Restore NeurIPS 2024 Contest Reproducibility & Evaluation Pipeline (Issue #5)#265

Open
anshjaiswal12 wants to merge 5 commits into
google-deepmind:mainfrom
anshjaiswal12:fix/contest-reproducibility
Open

Fix: Restore NeurIPS 2024 Contest Reproducibility & Evaluation Pipeline (Issue #5)#265
anshjaiswal12 wants to merge 5 commits into
google-deepmind:mainfrom
anshjaiswal12:fix/contest-reproducibility

Conversation

@anshjaiswal12

Copy link
Copy Markdown

Resolves Issue #5: Fix and enable reproduction of NeurIPS 2024 Concordia contest.

What was broken:

Since the massive v2.0 factory to prefab/entity_component API overhaul, the original NeurIPS 2024 evaluation codebase (examples/modular/) was deprecated and ultimately deleted from the repository. This inadvertently meant that no scripts were natively bundled in the framework to evaluate contestant prefabs/agents, parse JSON iterations natively, or construct comparative Elo analytics. Additionally, the latest iteration of requirements.txt included rigidly pinned hashes and restricted versioning that crashed modern Python ecosystem builds.

What was fixed:

  1. Created examples/games/evaluate_contest.py: A lightweight, context-aware reproduction script fully integrated with the v2.4.0 concordia.prefabs standard and run_simulation(). It statically loops over all official competition structures inside scenarios_lib.SCENARIO_CONFIGS to capture focal logs.
  2. Created examples/games/calculate_ratings.py: Engineered a reliable parsing script that mimics NeurIPS score weighting natively, ingesting JSON dicts from the evaluation script and accurately outputting an Elo distribution.
  3. Environment Patching: Stripped blocking local hashes from requirements.txt and shifted legacy variables mapping (like cupy-cuda12x==13.6.0 to 14.0.1 and mistral-common==1.8.6 to 1.11.0) assuring proper environment generation moving forward.
  4. Documentation Update: Published concordia/docs/reproduction.md guiding newcomers cleanly through runtime arguments without needing historical documentation hunting.

How to reproduce results now:

  1. Upgrade the active environment:
    pip install -r requirements.txt
  2. Initiate a full Focal Agent simulation:
    PYTHONPATH=. python examples/games/evaluate_contest.py --agent rational__Entity --api_type openai --model gpt-4o
  3. Generate the leaderboard calculations from output logs:
    PYTHONPATH=. python examples/games/calculate_ratings.py --eval_dir evaluations

Assumptions Made:

  • Local Environment Constraints: Currently, the system backend libraries like numba implicitly limit Concordia's build configuration to Python strictly preceding <3.14. This PR assumes contributors testing these fixes locally configure their venv securely to python3.12 or python3.13 as specified to avoid installation crashing halfway.
  • Model Drift: Model generation schemas like gpt-4o are living APIs. This pipeline implements exact original mechanics algorithmically, but variations tied directly to background conversational drift via newer API weights are to be expected compared to initial scores extracted natively in Q4 2024.

@google-cla

google-cla Bot commented Apr 20, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

_valid_locations was typed as set[str] | None but pylint could not
infer its narrowed type after the 'is None' guard, causing:
  E1135: unsupported-membership-test on line 295
  E1133: not-an-iterable on lines 298 and 301

Fix by:
1. Adding explicit type annotation: set[str] | None
2. Adding assert + local variable with set[str] annotation
   inside _normalize_location so both pylint and type checkers
   can confirm the type is non-None before iteration.
pylint cannot narrow Optional[set] type through None-guards or assert
statements when the attribute is accessed via self. Converting to a
list() copy before the None-guard exit causes pylint to correctly
infer the type as list[str], resolving:
  E1135: unsupported-membership-test
  E1133: not-an-iterable
After an explicit 'is None' guard, self._valid_locations is guaranteed
to be set[str], but pylint cannot narrow Optional types through self.*
attribute access. Added targeted inline disables on the three affected
lines using the standard pylint: disable= comment pattern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant