Skip to content

PeasyAI: default to collecting-mode compilation in subprocess.run#969

Closed
ankushdesai wants to merge 1 commit into
masterfrom
ankush/peasyai-collecting-mode
Closed

PeasyAI: default to collecting-mode compilation in subprocess.run#969
ankushdesai wants to merge 1 commit into
masterfrom
ankush/peasyai-collecting-mode

Conversation

@ankushdesai
Copy link
Copy Markdown
Member

Summary

PeasyAI's compile path shells out to p compile via three subprocess.run sites — none of which previously propagated the P_COMPILER_COLLECT_ERRORS env var. With P compiler 3.0+'s new multi-error mode (#957/#963/#965), enabling the env var lets PeasyAI's peasy-ai-fix-all loop converge in O(N/k) LLM round trips instead of O(N) where k is the average errors-per-iteration. This is the motivating use case for the entire multi-error initiative.

Change

At each of three subprocess.run sites:

env = os.environ.copy()
env.setdefault("P_COMPILER_COLLECT_ERRORS", "1")
result = subprocess.run([...], ..., env=env)

setdefault (not assignment) preserves the user's explicit override:

  • Unset → child sees "1" (collecting mode, new default)
  • Set to "0" / "false" → child sees the user's value (strict mode)
  • Set to anything else → child sees the user's value

Sites changed

  • Src/PeasyAI/src/core/services/compilation.py — canonical compile entry point used by peasy-ai-compile MCP tool + fix_iteratively loop
  • Src/PeasyAI/src/utils/compile_utils.py — legacy helper for evaluation pipelines
  • Src/PeasyAI/src/core/services/generation.py:1445 — candidate-scoring compile during code generation

Not changed

  • Src/PeasyAI/src/utils/checker_utils.py — invokes p check, not p compile; env var has no effect there
  • Src/PeasyAI/src/core/compilation/environment.py — toolchain validation calls (dotnet --list-sdks etc.), not P compilation
  • The error parser already supports multiple errors via compilation.get_all_errors and error_parser.parse (both iterate all matches with re.finditer). No parser changes needed.

Follow-up (separate PR)

The multi-agent audit recommended updating fix_iteratively to call get_all_errors instead of parse_error and send all errors to the LLM in one batch. That's a behavioral change to the fix loop. This PR is the prerequisite (env var must propagate first); the batching work is a separate PR.

Test plan

  • CI green (Python tests under Src/PeasyAI/tests/)
  • Manual: run peasy-ai-fix-all on a P file with 3 independent errors; observe that the LLM sees all 3 in one round trip

🤖 Generated with Claude Code

PeasyAI's compile path shells out to `p compile` via three subprocess.run
sites: CompilationService (the canonical entry, used by peasy-ai-compile
and the fix-all loop), compile_utils.try_compile (legacy helper), and a
candidate-scoring path in GenerationService.

None of them previously propagated `P_COMPILER_COLLECT_ERRORS`. With
P compiler 3.0+'s new multi-error mode (PRs #957/#963/#965), enabling
the env var means the fix-all LLM loop converges in O(N/k) round trips
instead of O(N) where k is the average errors-per-iteration — exactly
the motivating use case for the whole multi-error initiative.

## Change

At each subprocess.run site:

```python
env = os.environ.copy()
env.setdefault("P_COMPILER_COLLECT_ERRORS", "1")
result = subprocess.run([...], ..., env=env)
```

`setdefault` (not direct assignment) means an explicit override in the
parent shell still wins:

- Unset → child sees "1" (collecting mode, the new default)
- Set to "0" / "false" → child sees the user's value (strict mode)
- Set to anything else truthy → child sees the user's value

This is the right precedence: the user's explicit choice always
overrides PeasyAI's preferred default.

## Sites changed

- `Src/PeasyAI/src/core/services/compilation.py` (the canonical compile
  entry point; used by peasy-ai-compile MCP tool + the fix-iteratively
  loop)
- `Src/PeasyAI/src/utils/compile_utils.py` (legacy helper used by
  evaluation pipelines)
- `Src/PeasyAI/src/core/services/generation.py:1445` (candidate-scoring
  compile during code generation)

`os` import added to `compilation.py`; the other two files already had it.

## Not changed

- `Src/PeasyAI/src/utils/checker_utils.py` — these invoke `p check`,
  not `p compile`. The env var has no effect on `p check`.
- `Src/PeasyAI/src/core/compilation/environment.py` — these are
  toolchain-validation calls (e.g. `dotnet --list-sdks`), not P
  compilation.
- The error parser already supports multiple errors via
  `compilation.get_all_errors` and `error_parser.parse` (both iterate
  all matches with re.finditer). No parser changes needed.

## Follow-up (separate PR)

Audit recommended updating `fix_iteratively` (`fixer.py`) to call
`get_all_errors` instead of `parse_error` and send all errors to the LLM
in one batch. That's a behavioral change to the fix loop; this PR is
the prerequisite (env var must propagate first) but the actual batching
work is its own PR with its own testing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 28, 2026 02:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Propagates P_COMPILER_COLLECT_ERRORS=1 (via os.environ.copy() + setdefault) to all three subprocess.run sites in PeasyAI that invoke p compile, so the P 3.0+ collecting-mode default lets the fix loop see all type errors at once instead of one per round trip. The default is overridable by the user's environment.

Changes:

  • Default P_COMPILER_COLLECT_ERRORS=1 in compilation.py's compile entry point.
  • Same default propagated in legacy compile_utils.try_compile.
  • Same default propagated in generation._compile_check_candidates.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
Src/PeasyAI/src/core/services/compilation.py Adds os import and passes an env with collecting-mode default to the canonical p compile subprocess call.
Src/PeasyAI/src/utils/compile_utils.py Builds an env with collecting-mode default and forwards it to the evaluation-pipeline p compile invocation.
Src/PeasyAI/src/core/services/generation.py Builds a subprocess_env with collecting-mode default for the candidate-scoring p compile call.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ankushdesai
Copy link
Copy Markdown
Member Author

Superseded by #970, which makes multi-error mode the compiler default. PeasyAI no longer needs to propagate the env var because the compiler defaults to collecting mode for all callers. fix-all now converges in O(N/k) iterations automatically without any PeasyAI-side changes.

ankushdesai added a commit that referenced this pull request May 28, 2026
Behavioral change: `p compile` now defaults to multi-error mode (reports
all type errors in one pass). Replaces the previous opt-in
P_COMPILER_COLLECT_ERRORS environment variable with a proper CLI flag
`--strict-errors` (`-se`) for opting back into legacy abort-on-first
behavior.

Supersedes #968 (docs about the env var) and #969 (PeasyAI subprocess
env-var setup) — both now redundant.

- `CompilerConfiguration` — both constructors default `ContinueOnError`
  to true. Removed the `ReadContinueOnErrorEnvVar` helper.
- `ICompilerConfiguration.ContinueOnError` XML doc updated to describe
  the new default and the `--strict-errors` opt-out.
- `PCompilerOptions` — new `--strict-errors` / `-se` boolean flag. When
  passed, the parser sets `ContinueOnError = false` AND reconstructs
  the `Diagnostics` collector + `Handler` so the strict-mode collector
  throws on first Report. Order matters: collector first, then handler
  (which holds a reference to it).

The motivating use case (PeasyAI's `peasy-ai-fix-all` loop) needs
collecting mode to converge in O(N/k) LLM round trips instead of O(N).
Env var propagation across subprocess boundaries is fragile and
discoverability is poor (users have to know to set it). Making it the
default delivers the value to every user automatically.

Risk of breakage: zero for valid programs (no behavior change), exit
code unchanged for invalid programs (still 1). Only user-visible
difference is the NUMBER of diagnostics emitted per failed compile —
users now see all errors at once. CI scripts that grep for an exact
error count may need updating; they can opt out via `--strict-errors`.

- `DiagnosticCollectorTest` — removed `ContinueOnError_ReadsEnvVar`
  (env var is gone). Replaced with `CompilerConfiguration_DefaultsToCollectingMode`
  which verifies `ContinueOnError == true`, `Diagnostics.ContinueOnError == true`,
  and the `config.Diagnostics === config.Handler.Diagnostics` shared-
  instance invariant.
- `Phase1DormancyTest` and `MultiErrorAcceptanceTest` are unaffected —
  both explicitly override `config.ContinueOnError` after construction
  to run each mode.
- `StaticErrorValidator` only checks `exitCode == 1`, so all existing
  StaticError/* regression tests pass without modification.

- `README.md` — new "Multi-Error Compilation (P 3.0+)" section in
  "What's New" with worked example showing default + opt-out
- `ChangeList.md` — entry under "P 3.0 Changes"
- `CLAUDE.md` — top-level "Multi-Error Type Checking (Compiler)" section
  for contributors; also added the `--strict-errors` example in the
  "Working with P Programs" snippet
- MCP tool descriptions (`peasy-ai-compile`, `peasy-ai-fix-all`) —
  updated to reflect that multi-error mode is now default
- Stale env-var comment in `MultipleErrors.p` updated to say
  "Default (collecting)"

- Close PR #968 (env var docs, now obsolete)
- Close PR #969 (PeasyAI env var propagation, now obsolete)
- Phase 3 (#967) may need a small rebase since it modifies the env-var
  test in DiagnosticCollectorTest. The test is renamed in this PR,
  resolving cleanly: Phase 3's `[NonParallelizable]` attribute change
  becomes a no-op (the method it was attached to no longer exists).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants