Skip to content

fix(#21): Enforce strict parsing of LLM method output#47

Closed
AK11105 wants to merge 4 commits into
mainfrom
fix/21-parse-methods
Closed

fix(#21): Enforce strict parsing of LLM method output#47
AK11105 wants to merge 4 commits into
mainfrom
fix/21-parse-methods

Conversation

@AK11105
Copy link
Copy Markdown
Owner

@AK11105 AK11105 commented May 21, 2026

Summary

Fixes issue #21: _parse_methods conflates load() and predict() into one block.

Problem

The original implementation used lookahead regex ((?=\ndef |\Z)) to split methods from LLM output. This failed in three scenarios:

  1. Class-wrapped output: When the LLM wraps methods in a class despite the system prompt, both load_match and predict_match capture the entire class body, causing both methods to be identical.
  2. Missing blank lines: Without a blank line between methods, the lookahead \ndef doesn't match properly.
  3. Trailing text: Text after predict() with no subsequent def gets included in predict_body.

The bug manifests at runtime: self._model is never set in the _GeneratedModel class, raising AttributeError on first prediction.

Solution: Strict Parsing

Replace lookahead regex with explicit def boundary splitting AND enforce the system prompt contract:

blocks = re.split(r"(?=^def )", raw, flags=re.MULTILINE)
# Reject any non-method content (class wrappers, trailing text)
# Enforce: "Return ONLY the two method bodies as plain Python"

This approach:

  • Splits on start-of-line def — Rejects indented methods (class wrappers)
  • Validates no trailing content — Each block must be a valid method definition
  • Raises clear errors — Non-compliant output triggers LLM retry via fix()
  • Prevents silent failures — Better to fail validation than silently corrupt self._model

Why Strict?

If we accept edge cases (class wrappers, trailing text), the LLM learns it can ignore the system prompt. Strict parsing enforces the contract, preventing more bugs downstream.

Testing

  • test_parse_methods_no_blank_line_betweenAccepts (valid: missing blank line)
  • test_parse_methods_class_wrapped_raisesRejects (violates: no class wrapper)
  • test_parse_methods_trailing_text_raisesRejects (violates: ONLY two methods)

All existing tests continue to pass.

AK11105 added 4 commits May 21, 2026 12:47
- Split on ^def boundaries using MULTILINE flag instead of lookahead
- Avoids conflating methods when LLM wraps output in class body
- Handles missing blank lines between methods
- Properly isolates trailing text after predict()
- Validates both load and predict are found, raising clear error if not

This fixes:
1. Class-wrapped output where both methods get captured together
2. Missing blank line handling (no blank line = no \ndef boundary)
3. Trailing text after predict being included in the method body

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"
Add tests for the three main issues the fix addresses:
- Class-wrapped output (LLM wraps methods in class despite prompt)
- Missing blank line between methods
- Trailing text after predict method

These tests verify that the new explicit boundary splitting approach
correctly handles all edge cases without breaking existing functionality.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>"
Split on explicit def boundaries and reject non-compliant output.
Prevents conflating methods when LLM violates system prompt
(e.g., class wrapper, missing blank lines, trailing text).

Raises clear error to trigger LLM retry instead of silent failures.

- Split on ^def (start-of-line only, rejects indentation)
- Reject trailing content after predict
- Validate exactly two methods with correct signatures
- Error messages direct to system prompt requirements
Replace lenient edge-case tests with strict validation tests:
- test_parse_methods_class_wrapped_raises: Rejects indented methods (class wrapper)
- test_parse_methods_trailing_text_raises: Rejects trailing content

Keep test_parse_methods_no_blank_line_between: Valid format with missing blank line.

Enforces system prompt contract: "Return ONLY the two method bodies as plain Python"
@AK11105 AK11105 changed the title fix(#21): Replace lookahead regex with explicit def boundary splitting fix(#21): Enforce strict parsing of LLM method output May 21, 2026
@AK11105 AK11105 closed this May 21, 2026
@AK11105 AK11105 deleted the fix/21-parse-methods branch May 21, 2026 07:39
@AK11105
Copy link
Copy Markdown
Owner Author

AK11105 commented May 21, 2026

Copilot reeks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant