ci: add Makefile and workflow for agent script tests

[ifireball]

Summary

• Add a slim Makefile with script-test and test targets, porting the run-timed helper from fullsend
• Add .github/workflows/script-test.yml to run all seven existing scripts/*-test.sh suites on push, pull_request, and merge_group
• Install jsonschema in CI for validate-output-schema-test.sh

Linting remains in scope of #29; this PR focuses only on agent script test CI.

Test plan

• make script-test passes locally
• CI script-test workflow runs green on this PR

Made with Cursor

[ifireball]

Review requested from @fullsend-ai/core (team review could not be assigned via API — repo team access may need to be configured).

[qodo-code-review]

PR Summary by Qodo

CI: add Makefile target and GitHub workflow for agent script tests

⚙️ Configuration changes ✨ Enhancement 🕐 Less than 10 minutes

AI Description

• Add a Makefile target to run all agent shell script test suites with timing logs
• Add a GitHub Actions workflow to run script tests on push/PR/merge queue
• Install jsonschema in CI to support schema-validation test coverage

Diagram

graph TD
  A["GitHub Actions: script-test.yml"] --> B["make script-test"] --> C["run-timed macro"] --> D["scripts/*-test.sh"]
  A --> E["Python 3.12 + uv"] --> F["jsonschema"] --> D

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Run each scripts/*-test.sh directly in the workflow

• ➕ Removes Makefile dependency for CI
• ➕ Keeps all CI logic in one place
• ➖ Harder to run the exact same sequence locally
• ➖ More verbose workflow and duplicated command wiring

2. Use a GitHub Actions matrix to parallelize test suites

• ➕ Faster wall-clock CI time as suites grow
• ➕ Clear per-suite pass/fail reporting
• ➖ More workflow complexity (matrix, aggregation)
• ➖ Potentially repeats setup steps unless optimized

3. Single bash runner script (e.g., scripts/run-script-tests.sh)

• ➕ Keeps orchestration close to the scripts
• ➕ Portable across CI systems without Make
• ➖ Another custom harness to maintain
• ➖ Less discoverable than standard make targets for many repos

Recommendation: The current approach (Makefile target + CI workflow calling it) is a good default: it keeps CI thin, improves local reproducibility, and adds lightweight timing diagnostics. Consider moving to a matrix workflow only if runtime becomes a bottleneck.

Files changed (2) +56 / -0

Other (2) +56 / -0

script-test.ymlAdd CI workflow to run agent script tests +29/-0

Add CI workflow to run agent script tests

• Introduces a GitHub Actions workflow that runs on push, pull_request, and merge_group for main. It sets up Python/uv, installs jsonschema, and executes 'make script-test'.

.github/workflows/script-test.yml

MakefileAdd script-test target with timed execution wrapper +27/-0

Add script-test target with timed execution wrapper

• Adds 'script-test' and 'test' targets to run the seven existing shell test suites. Includes a 'run-timed' helper to emit per-suite duration while preserving exit codes.

Makefile

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:12 AM UTC · Completed 7:20 AM UTC
Commit: e282f02 · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. Protected .github/ workflow changed ✗ Dismissed 📜 Skill insight § Compliance

Description

This PR adds a new workflow under .github/workflows/, which is a protected
governance/infrastructure path requiring explicit human review and must not be auto-approved. A
compliance finding is required whenever protected paths are modified.

Code

.github/workflows/script-test.yml[R1-29]

+name: Script tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  merge_group:
+
+permissions:
+  contents: read
+
+jobs:
+  script-test:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.12"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
+
+      - name: Install test dependencies
+        run: uv pip install --system jsonschema
+
+      - run: make script-test

Evidence

The checklist explicitly enumerates .github/ as a protected path and requires raising a finding
for any PR that modifies such files. The diff adds .github/workflows/script-test.yml, triggering
this requirement.

.github/workflows/script-test.yml[1-29]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR modifies a protected governance/infrastructure path (`.github/workflows/`), which requires explicit human review and should include an explicit justification reference (e.g., linked issue/ADR) to support the change.

## Issue Context
Protected-path modifications are high impact because they can affect CI behavior and repository security posture.

## Fix Focus Areas
- .github/workflows/script-test.yml[1-29]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Broken run-timed macro ✗ Dismissed 🐞 Bug ≡ Correctness

Description

run-timed expands into multiple Make recipe lines with trailing \ continuations, but the
Makefile does not enable .ONESHELL, so each expanded line runs in a separate shell and the first
line is likely to fail with a shell parse error. This breaks make script-test, which the new
GitHub Actions workflow invokes.

Code

Makefile[R10-15]

+define run-timed
+	@start=$$(date +%s); \
+	rc=0; $(1) || rc=$$?; \
+	elapsed=$$(($$(date +%s) - $$start)); \
+	printf '::debug::script-test timing: %s completed in %ds\n' '$(1)' "$$elapsed"; \
+	exit $$rc

Evidence

The run-timed macro body contains multiple lines ending with \ and there is no .ONESHELL
directive in the Makefile; therefore Make will execute each expanded line in a separate shell and
the continuation will not work. The new workflow directly runs make script-test, so this bug will
surface in CI.

Makefile[1-16]
.github/workflows/script-test.yml[13-29]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`run-timed` is defined as a multi-line recipe fragment that relies on shell line-continuations (`\`). In GNU Make, each *recipe line* is executed in a separate shell by default; when the macro expands into multiple recipe lines, the trailing `\` cannot continue into the next shell invocation, causing failures when running `make script-test`.

### Issue Context
The CI workflow runs `make script-test`, so this will break the newly added script-test CI job as well.

### Fix Focus Areas
- Makefile[1-16]

### Suggested fix approaches (pick one)
1) **Rewrite `run-timed` into a single recipe line** (preferred):
  - Put the whole timing wrapper on one line separated by `;` so it runs in one shell.
2) **Enable `.ONESHELL:`** near the top of the Makefile *and* ensure command echoing/`set -e` behavior is what you want (e.g., add leading `@` to each line or set shell flags).

After the change, confirm `make script-test` runs all scripts and correctly returns the failing script’s exit code.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. ::debug:: prints unsanitized $(1) ✗ Dismissed 📜 Skill insight ⛨ Security

Description

The new run-timed helper emits a GitHub Actions workflow command (::debug::) while interpolating
$(1) and $$elapsed without sanitizing for :: sequences, encoded newlines (%0A/%0D), ANSI
escapes, or control characters. This can enable workflow-command injection/log annotation
manipulation if any interpolated value becomes attacker-controlled now or in future reuse.

Code

Makefile[14]

+	printf '::debug::script-test timing: %s completed in %ds\n' '$(1)' "$$elapsed"; \

Evidence

The compliance rule requires every interpolated value used in a GitHub Actions workflow command to
be individually sanitized. The added line prints ::debug::... and interpolates both $(1) and
$$elapsed without any sanitization step.

Makefile[10-15]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`Makefile` emits a GitHub Actions workflow command via `printf '::debug::...'` while interpolating values that are not individually sanitized, which violates the requirement for safe GHA command construction.

## Issue Context
Any `::debug::` / `::warning::` / `::error::`-style annotation is parsed by the GitHub Actions runner. All interpolated fields must be sanitized to prevent command injection via `::`, `%0A/%0D`, ANSI escapes, or control characters.

## Fix Focus Areas
- Makefile[10-15]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

Review

Clean CI infrastructure PR that adds a Makefile and GitHub Actions workflow to run the repo's seven existing script test suites. No correctness, security, or architectural concerns. The second commit (e40f422c) addresses the prior review's supply-chain finding by pinning jsonschema==4.23.0.

Dimension	Result
Correctness	✅ All 7 test scripts exist and match Makefile references. run-timed macro correctly propagates exit codes via rc=0; CMD || rc=$?; exit $rc. Make halts on first failure.
Security	✅ Minimal permissions (contents: read). All actions SHA-pinned. pull_request trigger (not pull_request_target). ::debug:: annotation uses only hardcoded string literals and numeric values — no injection risk. jsonschema now pinned to ==4.23.0.
Intent & coherence	✅ PR title follows conventional commits (ci: prefix). Scope matches stated intent — CI infrastructure for existing test scripts, no creep.
Style & conventions	✅ Workflow structure (SHA-pinned actions with version comments, ubuntu-24.04, top-level permissions:) matches the repo-owned release.yml pattern. .yml extension is consistent with repo-owned workflow convention.
Documentation	⚠ Minor gaps — see low-severity findings below.
Cross-repo contracts	⏭ Skipped — no exported interfaces or APIs modified.

Findings

High

• [protected-path] .github/workflows/script-test.yml — This file is under the .github/ protected path. The PR does not link to a tracking issue that authorizes the change. The PR description adequately explains the rationale (CI infrastructure for existing test scripts), but protected-path changes require both a linked issue and human approval.
  Remediation: Link a tracking issue to provide authorization context, or obtain direct human reviewer approval.

Low

• [docs-staleness] README.md:49 — The Workflows table does not list the new script-test.yml workflow. After this PR merges, the table will be incomplete.
  Remediation: Add a row: | script-test.yml | This repo | Runs agent shell script tests on PRs and main branch pushes |

• [docs-staleness] README.md — The README has no development/testing section. The PR introduces make test / make script-test as the standard way to run tests, but contributors may not discover it.
  Remediation: Add a brief "Testing" section, e.g., "Run make test to execute all script test suites."

Previous run

Review

Verdict: approve ✅

Clean CI infrastructure PR that adds a Makefile and GitHub Actions workflow to run the repo's seven existing script test suites. No correctness, security, or architectural concerns.

What was reviewed

Dimension	Result
Correctness	✅ All 7 test scripts exist and match Makefile references. run-timed macro correctly propagates exit codes (Make halts on first failure). jsonschema dependency verified as required by validate-output-schema.sh.
Security	✅ Minimal permissions (contents: read). All actions SHA-pinned. pull_request trigger (not pull_request_target) — no secret exposure to forks. ::debug:: annotation uses only hardcoded/numeric values, no injection risk.
Intent & coherence	✅ No linked issue, but the change is self-evidently authorized CI infrastructure for existing test scripts. Scope matches PR description exactly — no creep. PR title follows conventional commits format (ci: prefix).
Style & conventions	✅ Workflow structure (SHA-pinned actions with version comments, ubuntu-24.04, top-level permissions:) matches the repo-owned release.yml pattern. .yml extension is consistent with release.yml (the .yaml file is externally managed by fullsend).
Documentation	⚠ Minor gaps — see low-severity findings below.
Cross-repo contracts	⏭ Skipped — no exported interfaces or APIs modified.

Findings

3 low-severity findings (non-blocking)

1. supply-chain — .github/workflows/script-test.yml:28

jsonschema is installed from PyPI without a version pin (uv pip install --system jsonschema). A compromised package update would execute in the CI runner. Blast radius is limited by the read-only GITHUB_TOKEN, but pinning to a known-good version is best practice.

Remediation: Pin to a specific version, e.g., uv pip install --system jsonschema==4.23.0, or use a requirements file with hash verification.

2. docs-staleness — README.md:49

The Workflows table explicitly documents all workflow files but will become incomplete after this PR merges. The new script-test.yml workflow is not listed.

Remediation: Add a row: | script-test.yml | This repo | Runs agent shell script tests on PRs and main branch pushes |

3. docs-staleness — README.md

The PR introduces make test / make script-test as the standardized way to run tests, but the README has no testing or development section. The Makefile's help target provides self-documentation, but contributors may not discover it.

Remediation: Consider adding a brief "Testing" section to the README, e.g., "Run make test to execute all script test suites."

---

Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

• .github/workflows/script-test.yml

[ifireball]

Babysit triage of review threads:

#3 Broken run-timed / .ONESHELL — Not reproducible. $(call run-timed,...) expands to a single recipe with \ continuations (one shell per invocation). Verified locally (make script-test) and in CI (script-test job, 32s, green).

#1 ::debug:: sanitization — $(1) is only hardcoded bash scripts/*-test.sh literals at call sites (ported unchanged from fullsend). No attacker-controlled input path.

#2 Protected .github/ workflow — Acknowledged; this PR intentionally adds script-test CI. Human review from @fullsend-ai/core requested.

jsonschema pinning — Addressed in follow-up commit (jsonschema==4.23.0, matching fullsend sandbox).

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:38 AM UTC · Completed 7:47 AM UTC
Commit: e40f422 · View workflow run →

[fullsend-ai-review]

See the review comment for full details.

[ralphbean]

LGTM.