Skip to content

Add performance test size specifications to all challenge specs#165

Merged
kunal-mansukhani merged 3 commits intomainfrom
copilot/update-test-case-size-specs
Feb 4, 2026
Merged

Add performance test size specifications to all challenge specs#165
kunal-mansukhani merged 3 commits intomainfrom
copilot/update-test-case-size-specs

Conversation

Copy link
Contributor

Copilot AI commented Feb 3, 2026

Challenge specs were missing performance test size information, making it unclear what dimensions are used for benchmarking. Only challenge 2 (matrix multiplication) documented this.

Changes

  • Added performance test size bullets to Constraints section in 65 challenge specs
  • Automated extraction via AST parsing of generate_performance_test() methods
  • Consistent formatting: <code> tags for parameters, comma separators for values ≥ 1,000
  • Updated CLAUDE.md to document the new spec format requirement for future challenges

Examples

Single parameter:

<li>Performance is measured with <code>N</code> = 25,000,000</li>

Multiple parameters:

<li>Performance is measured with <code>M</code> = 8,192, <code>N</code> = 6,144, <code>K</code> = 4,096</li>

All 66 challenges now explicitly document their performance test dimensions, and the challenge creation guide has been updated to require this for all new challenges.

Original prompt

To the spec add a bullet that says the test case size for the performanc test just like we do for challenge 2 matrix multiplication. Add this for all challenges


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: kunal-mansukhani <24417742+kunal-mansukhani@users.noreply.github.com>
Copilot AI changed the title [WIP] Add test case size for performance tests Add performance test size specifications to all challenge specs Feb 3, 2026
@kunal-mansukhani kunal-mansukhani marked this pull request as ready for review February 3, 2026 22:58
@kunal-mansukhani
Copy link
Contributor

@copilot Update CLAUDE.md to make note of this new change in spec format

Co-authored-by: kunal-mansukhani <24417742+kunal-mansukhani@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 3, 2026

@copilot Update CLAUDE.md to make note of this new change in spec format

Updated CLAUDE.md to document the performance test size requirement in the challenge.html section and checklist (commit 24bd90b).

@kunal-mansukhani kunal-mansukhani merged commit 5f9ce80 into main Feb 4, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants