Fix: Restore NeurIPS 2024 Contest Reproducibility & Evaluation Pipeline (Issue #5)#265
Open
anshjaiswal12 wants to merge 5 commits into
Open
Fix: Restore NeurIPS 2024 Contest Reproducibility & Evaluation Pipeline (Issue #5)#265anshjaiswal12 wants to merge 5 commits into
anshjaiswal12 wants to merge 5 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
_valid_locations was typed as set[str] | None but pylint could not infer its narrowed type after the 'is None' guard, causing: E1135: unsupported-membership-test on line 295 E1133: not-an-iterable on lines 298 and 301 Fix by: 1. Adding explicit type annotation: set[str] | None 2. Adding assert + local variable with set[str] annotation inside _normalize_location so both pylint and type checkers can confirm the type is non-None before iteration.
pylint cannot narrow Optional[set] type through None-guards or assert statements when the attribute is accessed via self. Converting to a list() copy before the None-guard exit causes pylint to correctly infer the type as list[str], resolving: E1135: unsupported-membership-test E1133: not-an-iterable
After an explicit 'is None' guard, self._valid_locations is guaranteed to be set[str], but pylint cannot narrow Optional types through self.* attribute access. Added targeted inline disables on the three affected lines using the standard pylint: disable= comment pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves Issue
#5: Fix and enable reproduction of NeurIPS 2024 Concordia contest.What was broken:
Since the massive v2.0
factorytoprefab/entity_componentAPI overhaul, the original NeurIPS 2024 evaluation codebase (examples/modular/) was deprecated and ultimately deleted from the repository. This inadvertently meant that no scripts were natively bundled in the framework to evaluate contestant prefabs/agents, parse JSON iterations natively, or construct comparative Elo analytics. Additionally, the latest iteration ofrequirements.txtincluded rigidly pinned hashes and restricted versioning that crashed modern Python ecosystem builds.What was fixed:
examples/games/evaluate_contest.py: A lightweight, context-aware reproduction script fully integrated with thev2.4.0concordia.prefabsstandard andrun_simulation(). It statically loops over all official competition structures insidescenarios_lib.SCENARIO_CONFIGSto capture focal logs.examples/games/calculate_ratings.py: Engineered a reliable parsing script that mimics NeurIPS score weighting natively, ingesting JSON dicts from the evaluation script and accurately outputting an Elo distribution.requirements.txtand shifted legacy variables mapping (likecupy-cuda12x==13.6.0to14.0.1andmistral-common==1.8.6to1.11.0) assuring proper environment generation moving forward.concordia/docs/reproduction.mdguiding newcomers cleanly through runtime arguments without needing historical documentation hunting.How to reproduce results now:
pip install -r requirements.txtPYTHONPATH=. python examples/games/evaluate_contest.py --agent rational__Entity --api_type openai --model gpt-4oPYTHONPATH=. python examples/games/calculate_ratings.py --eval_dir evaluationsAssumptions Made:
numbaimplicitly limit Concordia's build configuration to Python strictly preceding<3.14. This PR assumes contributors testing these fixes locally configure theirvenvsecurely topython3.12orpython3.13as specified to avoid installation crashing halfway.gpt-4oare living APIs. This pipeline implements exact original mechanics algorithmically, but variations tied directly to background conversational drift via newer API weights are to be expected compared to initial scores extracted natively in Q4 2024.