Reused temp dir across validation retries causes stale module state and non-deterministic failures

[AK11105]

What happened?

validate_pipeline is called inside a loop that reuses the same tmp_dir across all retry attempts. The function pops sys.modules["definition"] after each run, but the .py file written to tmp_dir persists. If a background thread holds a reference to the previous import during cleanup, the module loader picks up the stale file on the next attempt.

This produces a confusing pattern: attempt 2 passes, attempt 3 fails with the same error as attempt 1.

# current deploy.py
with tempfile.TemporaryDirectory() as tmp_dir:
    for attempt in range(1, _MAX_RETRIES + 1):
        result = validate_pipeline(source, answers.sample_input, Path(tmp_dir))

Steps to reproduce

1. Deploy a model where the first codegen attempt produces broken code
2. Observe attempt 2 passes validation
3. Observe attempt 3 (if triggered) fails with the same error as attempt 1
4. The written scaffold may be from attempt 2 or 3 depending on timing

Expected behavior

Create a fresh subdirectory per attempt:

with tempfile.TemporaryDirectory() as tmp_root:
    for attempt in range(1, _MAX_RETRIES + 1):
        tmp_dir = Path(tmp_root) / str(attempt)
        tmp_dir.mkdir()
        result = validate_pipeline(source, answers.sample_input, tmp_dir)

Cost is negligible (three small directories). Eliminates the stale-module class of failures entirely.

Environment

• Area: app/cli/commands/deploy.py
• Phase: 8 (fix before Phase 9 begins)
• Priority: Low

Relevant logs or error output

Attempt 1/3 failed: <error>
Attempt 2/3 passed
Attempt 3/3 failed: <same error as attempt 1>