P0.3 public-demo advisory release-evidence rehearsal

[pengfei-threemoonslab]

Run type: public-demo rehearsal, not design-partner validation
Reviewer: @pengfei-threemoonslab
Review slot: 2026-05-20 15:48:14 PDT / immediate review
Shipgate ref: 5ba36d9
Target SHA: 9514473c234c8419b812b658157a5c3d4341713f

Target

• Repo: https://github.com/openai/openai-agents-python
• Primary target: examples/customer_service
• Fallback target: another OpenAI Agents SDK example only if the primary target is evidence-empty
• Framing: public-demo rehearsal only; not upstream owner validation and not design-partner validation.

Run Notes

To be filled during execution:

• Exact target SHA: 9514473c234c8419b812b658157a5c3d4341713f
• Setup timing, cold cache: 15 seconds for clone + venv + pinned Shipgate install
• Warm-cache under 10 minutes: yes on this machine; first setup completed under 10 minutes
• First-run failures: none at command/runtime level; advisory scan exited 0. Release decision is insufficient_evidence due to low-confidence extracted tools.
• Manifest edits: generated examples/customer_service/shipgate.yaml, replaced placeholders with airline customer service agent name and purpose; did not run --agent-instructions.
• Scan decision: insufficient_evidence
• Blockers / review items: 0 blockers, 3 review items
• Privacy audit enabled: true
• Privacy redacted occurrence count: 0
• Privacy output surfaces: json, packet_html, packet_json, packet_md
• False positives: pending reviewer review
• Missing checks: pending reviewer review
• Confusing wording: packet §1 says insufficient_evidence due to 2 low-confidence tools, but packet §10 residuals says low-confidence tool extractions: none.
• Install friction: v0.10.0 tag absent; used pinned commit SHA. Pinned git install worked.
• Reviewer feedback: pending @pengfei-threemoonslab review
• Would advisory CI be acceptable: pending reviewer review; local advisory-only workflow draft prepared.

Reviewer Brief

This is a public-demo rehearsal, not owner validation. A virgin advisory run will look noisier than a maintained baseline-backed workflow. Advisory exit 0 means findings did not fail CI; config, parse, missing-source, and runtime errors are first-run failures and should be categorized separately.

Existing Benchmark Cross-Reference

This run coexists with benchmark/repos/openai-agents-sdk. It does not replace that fixture. If this rehearsal exposes repeated friction, create a follow-up to add or adjust a public-demo OpenAI SDK archetype.

[pengfei-threemoonslab]

P0.3 run update — local advisory scan complete

• Target SHA: 9514473c234c8419b812b658157a5c3d4341713f
• Setup timing: 15 seconds for clone + venv + pinned Shipgate install on this machine
• Target checkpoint: examples/customer_service contains direct @function_tool usage and update_seat
• Reports were written outside the target checkout because upstream .gitignore does not ignore agents-shipgate-reports/
• Manifest generated: examples/customer_service/shipgate.yaml; placeholders replaced; no --agent-instructions run
• Local advisory CI draft prepared at .github/workflows/agents-shipgate.yml with action pinned to 5ba36d90549086ca4b4c5b40a8261ac468d2acee

Release decision

• release_decision.decision: insufficient_evidence
• Reason: Evidence coverage below threshold (2 low-confidence tools); scan results are not trustworthy enough to gate release
• Advisory CI behavior: exit 0, would_fail_ci=false
• Blockers: 0
• Review items: 3
• Source warnings: none

Active findings

• SHIP-AUTH-MISSING-SCOPE (high), update_seat: write-class tool lacks declared auth scopes
• SHIP-SCHEMA-FREEFORM-OUTPUT (medium), faq_lookup_tool: returns free-form text output
• SHIP-SCHEMA-FREEFORM-OUTPUT (medium), update_seat: returns free-form text output

Privacy audit

• privacy_audit.enabled: true
• privacy_audit.redacted_occurrence_count: 0
• privacy_audit.output_surfaces: json, packet_html, packet_json, packet_md

Product friction

• The missing v0.10.0 tag is real install/CI friction; this run used the pinned commit instead.
• Packet wording inconsistency: §1 says insufficient_evidence due to 2 low-confidence tools, while §10 residuals says “Low-confidence tool extractions: none.” That should become a product follow-up if confirmed.

Reviewer framing

This is a public-demo rehearsal, not owner validation. A virgin advisory run will look noisier than a maintained baseline-backed workflow. The key reviewer question is whether the packet and the three findings make the review more concrete despite the insufficient-evidence verdict.

[pengfei-threemoonslab]

Product follow-ups created

• Packet residuals contradict insufficient_evidence low-confidence count #101 — Packet residuals contradict insufficient_evidence low-confidence count
• Release docs and advisory CI reference v0.10.0 before tag exists #102 — Release docs and advisory CI reference v0.10.0 before tag exists