test(v1.8.2): extend CLI smoke to real HTTP run + artifact checks#50
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e20dea1860
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // `.aqa/` artifacts it expects. Using a tmp dir keeps the smoke test | ||
| // hermetic — no checked-in fixture to drift out of date. | ||
| const SANDBOX = mkdtempSync(`${tmpdir()}/aqa-cli-e2e-`); | ||
| const APP_PORT = 3189; |
There was a problem hiding this comment.
Using a fixed APP_PORT = 3189 makes this e2e test environment-dependent: if that port is already occupied (for example by another local process or parallel job on a shared runner), app.listen(...) fails with EADDRINUSE and the smoke check fails for reasons unrelated to aqa run. This introduces avoidable flakiness in the new CI gate; bind to port 0 and propagate server.address().port into project.yaml so the test always uses an available local port.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR strengthens the CLI E2E smoke test by exercising a real aqa run --profile smoke against a locally hosted HTTP /healthz endpoint, and by asserting that run artifacts are written under .aqa/runs/<run-id>/. It also documents the completed v1.8.2 slice and captures the testing pattern as a lesson.
Changes:
- Extend
scripts/e2e-cli.mjsto create a temp sandbox, seed a minimal project/profile + local pack, start an in-process HTTP server, runaqa run --profile smoke, and assert artifact outputs. - Update progress/lesson docs to reflect the new “network-real” CLI smoke pattern and the v1.8.2 slice closure.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| scripts/e2e-cli.mjs | Adds in-process HTTP target, seeds a local smoke pack/profile, runs aqa run, and checks emitted artifacts. |
| docs/PROGRESS.md | Notes closing the v1.8.2 slice with the improved CLI smoke test. |
| docs/LESSON.md | Records the self-contained, network-real CLI smoke testing pattern for future reuse. |
| const SANDBOX = mkdtempSync(`${tmpdir()}/aqa-cli-e2e-`); | ||
| const APP_PORT = 3189; | ||
|
|
| await new Promise((resolve) => app.listen(APP_PORT, '127.0.0.1', () => resolve(undefined))); | ||
|
|
| // Verify `aqa run` produced run artifacts with a non-empty events chain. | ||
| const runsDir = join(SANDBOX, '.aqa', 'runs'); | ||
| if (failed === 0) { | ||
| const runIds = readdirSync(runsDir).sort(); | ||
| if (runIds.length === 0) { | ||
| failed += 1; | ||
| console.error('✗ run-smoke did not produce any .aqa/runs/<run-id> directory'); | ||
| } else { | ||
| const latest = join(runsDir, runIds[runIds.length - 1]); | ||
| const eventsPath = join(latest, 'events.jsonl'); | ||
| const findingsPath = join(latest, 'findings.jsonl'); | ||
| const eventsText = readFileSync(eventsPath, 'utf8').trim(); | ||
| const findingsText = readFileSync(findingsPath, 'utf8'); |
| const findingsText = readFileSync(findingsPath, 'utf8'); | ||
| if (!eventsText) { | ||
| failed += 1; | ||
| console.error('✗ run-smoke produced empty events.jsonl'); | ||
| } | ||
| if (typeof findingsText !== 'string') { |
| * 1. `aqa --version` — binary is wired | ||
| * 2. `aqa --help` — help text is reachable | ||
| * 3. `aqa doctor` — project profiler runs without throwing | ||
| * 4. `aqa validate` — schema-validates the example's | ||
| * agentic-qa-kit.yaml | ||
| * | ||
| * What it does NOT do (deferred): | ||
| * - `aqa run` against a live target. The runner contract is exercised | ||
| * in `packages/runner/test/*`; spinning up the example app + LLM | ||
| * adapter inside CI is a Task 7 follow-up. | ||
| * 4. `aqa validate` — schema-validates generated .aqa/* | ||
| * 5. `aqa run --profile smoke` against a live local HTTP target | ||
| * + verifies run artifacts are written |
Summary
aqa run --profile smokepathValidation