feat(ai-engineer): add agent-improver meta-engineer routine#2251
Conversation
The Daily AI Engineer improves the products; nothing improves the Engineer itself from outside a single run. The existing self-improvement skill is one run reflecting on its own memory, so it cannot see patterns that only appear across hundreds of runs or across the two sibling instances. Adds a daily meta-engineer that mines both deployed instances' own operational telemetry and improves their definition, loaders and permission layer: - .claude/scripts/agent-telemetry.sh — read-only miner over the Claude session corpus, Codex sessions/log DB, both memory stores, the loaders and GitHub outcomes. Emits one scorecard across reliability, efficiency, safety, cross-instance coordination, drift and outcomes. - .claude/agents/agent-improver.md — the meta-engineer: the ingestion boundary, the six-parameter scorecard, the authority model and the audit trail. - .claude/skills/agent-improvement/SKILL.md — gather → score → diagnose → act → verify → record, with the verify step that closes the loop. - .claude/scripts/agent-telemetry.test.sh — 19 behavioural tests on synthetic fixtures, including a regression guard for reading the sibling's cadence as the loader's own. The ingestion boundary is load-bearing: the corpus is saturated with untrusted content, so the miner extracts behaviour (counts, timings, error signatures) and never treats corpus prose as instruction. Only a measured pattern or the maintainer's direct session direction authorizes a change. Validated against the live corpus, which surfaced real defects on first run: both loaders still asserted the definition-PR promotion gate retired in #2239, the Claude loader misstated its own cadence, ~50 busy-wait attempts/day the latency discipline forbids, and a recurring `gh pr view` missing --repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every other script under .claude/scripts has a CI test job; without one the
new miner's 19 tests would rot unrun. Wires it in on both OSes — the daily
runs happen on macOS while CI defaults to Linux.
Also replaces the loader cadence extraction with portable awk. It used a
bounded lazy quantifier (`{0,160}?`), which is not portable ERE: macOS ugrep
accepted it, but GNU grep on the Ubuntu runner may not, and dropping the lazy
marker instead trips ugrep's complexity limit. awk's index() also fixes the
underlying problem more directly — it takes the FIRST "dispatched" after the
self-identifying clause, which matters because the Codex loader packs both
its own and the sibling's cadence onto one line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both sides added a paths-filter output and a test job. Kept both, and adopted main's self-gating pattern (gate the filter on ci.yaml itself) for the agent-telemetry filter too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d26a7d218d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Eight findings, all real. Two were security defects in a script whose whole job is to be trustworthy about safety. P1 redaction: the reliability section printed the first 100 chars of raw error text, and a failed tool result routinely carries the command that failed — which can carry a token. Redaction now lives at the output boundary rather than per-detector, so a forgotten call-site cannot leak. P1 authority: the agent definition claimed enforcement-layer write access in both directions, contradicting the contract's reservation of that layer — and of amendments to the reservation itself — to the maintainer's own keystroke. Split by layer: prose is symmetric per his 2026-07-18 direction; enforcement tightening is direct, enforcement widening is prepared and handed over. Self- granting that access would have been an amendment to the one rule forbidding it, and a control the agent removes on request constrains nobody. P2 structural denials: guard firings were counted by grepping raw transcript text, so untrusted prose quoting a denial phrase was counted as a real block. That is the exact input the improver uses to decide guard-vs-agent, so transcript content could manufacture evidence for loosening a guard. Now anchored to errored tool_result envelopes. P2 Codex coverage: detectors read only the Claude corpus while the script claimed to score both instances. The format-agnostic detectors (credentials, sleeps, timeouts) now cover both; tool-attributed reliability stays Claude- schema-only and says so, since Codex has no is_error equivalent — a known gap stated in the output rather than a silent overclaim. P2 fine-grained PATs: github_pat_ was missing, so a modern token leak reported "clean" — the worst failure mode for a leak detector. P3 arg validation: an option without its value spun the parse loop forever. P3 ordering: find|head took directory order, not newest-first, so the cap could drop the newest failures and skew the scorecard. P2 frontmatter: `tools: All tools` is not a tool name; full access is the omitted default, matching every other agent here. Adds 8 regression guards (27 tests total). Each finding was reproduced RED first — every one of them failed silently in a way that looked clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex review — resolution record (all 8 findings, fixed in f743cb5)The inline replies landed thin because the push outdated their line anchors, so here is the full record. All eight were valid; two were security defects in a script whose entire job is to be trustworthy about safety. P1 — Preserve maintainer control over guardrails. The most important finding. The definition claimed enforcement-layer write access in both directions, contradicting the contract. I had raised this concern with the maintainer twice before building and he overrode it in session, so I recorded the grant as given — that was the wrong resolution. His direction is a valid channel for the prose layer, but the contract reserves the enforcement layer and any amendment to the rule reserving it to his own keystroke ("never apply it yourself"), and says the reservation "binds every amendment after it". Self-granting that access would have been an amendment to the one rule forbidding me from making it. Now split: prose symmetric (his direction, recorded with date); enforcement tightening applied directly, enforcement widening prepared, verified, explained, and handed over. Plus an explicit bar on routing around it by editing the prose rule that creates it. He can widen it whenever he likes — by editing that bullet himself. P1 — Redact secrets before printing error signatures. A failed tool result routinely carries the command that failed, and that command can carry a token; printing 100 raw characters turned a private transcript leak into a run-report leak. Redaction now sits at the output boundary rather than in one detector — a forgotten call-site is exactly how this happened — sharing its pattern set with the leak detector so the two cannot drift. P2 — Parse guard denials structurally. The one I'm most glad you caught. Blocked-action counts are what the improver uses to tell "the guard is wrong" from "the agent is wrong", so a raw text grep let untrusted transcript prose manufacture evidence for loosening a guard — the ingestion boundary failing precisely where it exists to protect. Now anchored to errored P2 — Include Codex sessions. Fixed partly, and the output now says which part. Format-agnostic detectors (credentials, sleeps, timeouts) now read both corpora. Tool-attributed reliability can't follow — Codex uses P2 — Detect fine-grained PATs. P3 — Reject missing option values. Confirmed your diagnosis: P3 — Sort before the cap. P2 — Omit the invalid all-tools allowlist. Confirmed against this repo: every other agent lists real tool names, and full-access agents omit the field entirely. Tests: 8 regression guards added (27 total). Each finding was reproduced RED first — every one failed silently in a way that looked like a clean result, which is what made them worth guarding rather than just fixing. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f743cb507c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex's second pass found seven more, and the useful signal is that most were the same classes as round 1 at call sites the first fix missed. Round 1 fixed instances; this fixes the classes. Redaction is now structural, not remembered. The previous commit claimed to move redaction "to the output boundary" but still required each detector to pipe through it — and the near-miss sampler did not, printing raw commands that carry inline env assignments like `GITHUB_TOKEN=… npm ci`. All output now flows through one `main | redact`, so a new detector is covered by construction rather than by review. Behavioural counts are now structural everywhere, not just for denials. Sleep counting still grepped raw text, so untrusted prose mentioning `sleep 60` fabricated busy-wait evidence — the same injection the denial fix closed, one metric over. Both now read only commands the agent actually ran, via a shared extractor covering the Claude and Codex schemas. Detector/redactor parity is now enforced by test rather than asserted. The previous commit claimed a shared pattern set "so they cannot drift"; they had already drifted — the redactor masked JWTs, the detector missed them, so a bearer-token leak reported clean. Section gating used the Claude count alone, so a Codex-only window reported "no sessions" while its format-agnostic leak and busy-wait detectors were skipped. Now gated on the combined count. GNU `stat -f` means --file-system and SUCCEEDS, so the failure-based fallback never fired on Linux and its output polluted the session list — one real file counted as several phantom paths, corrupting the scorecard on the CI/Ubuntu path that local macOS runs could not catch. Flavour is now detected once. The skill's routing table still sent permission changes to a direct edit, contradicting the agent's authority split — the table is the runbook, so it could have widened its own guard. Split into tighten-direct vs widen-handoff. Agent description no longer claims "full authority" either. 9 more guards (36 total), each per-class. One caught a vacuous assertion in my own test: the isolated-corpus fixture proved the sleep filter still counts a real sleep command rather than having stopped counting everything. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex round 2 — resolution record (7 findings, fixed in ac5483c)All valid. The useful signal is that most were the same classes as round 1, at call sites the first fix missed — round 1 fixed instances, so this round fixes the classes. P1 — Redact command near-miss samples. My round-1 message claimed redaction had moved "to the output boundary", but it hadn't: each detector still had to remember to pipe through it, and the near-miss sampler didn't — printing raw commands, which carry inline env assignments like P1 — Don't direct-edit enforcement widenings. The skill's routing table still sent permission changes to a direct edit, contradicting the authority split I'd just made in the agent definition. The table is the runbook the agent follows, so it could have widened its own guard — the exact outcome the split exists to prevent. Now two rows: tightening direct, widening prepared and handed over. P2 — Ignore transcript prose when counting sleeps. Same injection as the round-1 denial fix, one metric over. Both now read only commands the agent actually ran, through a shared extractor spanning the Claude and Codex schemas. Guarded both ways: prose mentioning P2 — Flag JWT-style tokens. Round 1 claimed detector and redactor shared a pattern set "so they cannot drift". They had already drifted — the redactor masked P2 — Codex-only windows (safety + efficiency). Both sections gated on the Claude count alone, so a Codex-only window reported "no sessions" while its format-agnostic detectors were skipped. Combined gate now. P2 — GNU stat output. The one I could not have caught locally: GNU Tests: 9 more guards, 36 total, written per-class rather than per-instance. One note worth recording: while resolving these threads, my own |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ac5483cb5f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…dings
Authority: the maintainer rejected the narrowing outright ("That was wrong.
You need symmetric authority"), so full symmetric authority is restored across
the agent, skill and loader, and AGENTS.md now carries an explicit carve-out
recording that direction. The carve-out is scoped to the improver as a distinct
actor — the daily engineer's own enforcement-layer reservation is untouched,
and neither actor may widen its OWN authority from telemetry. With the approval
gate gone, the loosening evidence bar is now the control that stands in its
place, so it is stated as load-bearing rather than procedural. The definition
also records that this narrowing was proposed twice and rejected twice, so a
future run does not re-derive a settled decision.
Eight review findings, of which two were recurrences of classes I had already
"fixed" twice — so they are fixed at the level that prevents recurrence:
- Raw error text was written to a predictable /tmp path before redaction. The
output-boundary wrap only covers stdout, so the scratch file leaked. Now
mktemp (0600), redacted on the way IN, removed on any exit including signals.
- Timeouts were still grepped from whole transcripts, so prose quoting
"Command timed out after 2m" inflated them — the third appearance of the
prose-fabrication class after denials and sleeps. Now read from real tool
results; the real-corpus count fell 35 -> 8, so the inflation was material.
- Detector/redactor parity broke a second time (generic `token=` assignments).
The test suite now asserts every credential shape is BOTH detected AND
redacted, so adding a shape to one list fails the build.
- Codex commands were never actually parsed. Real records are custom_tool_call
name="exec" whose .input is JavaScript (`tools.exec_command({cmd: "…"})`),
not the JSON `arguments` shape my fixture invented — so the fixture passed
for two rounds while the shipping format went unread. Now parsed from the
real shape, with a fixture copied from an actual session.
- Collision metrics read only the Claude corpus, which is self-defeating for a
cross-instance metric: the sibling half of the interaction was invisible.
- The near-miss detector flagged every npm ci/go generate, including our own
builds. A detector that always fires teaches you to ignore it. Now requires a
non-own checkout in the same session, and is labelled a candidate list.
- Merged-PR count capped at 30, which saturates on a weekly window and reads as
a real plateau. Raised to 300.
- "Never touch product repos" contradicted the authority section, which allows
editing a submodule's AGENTS.md Maintenance card. Scoped to product WORK; the
definition surface inside a product repo stays editable via a draft PR.
12 more guards (48 total).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex round 3 — resolution record (f0d0803)Eight of nine accepted. One refuted, with reasoning. Refuted — "Remove the blanket enforcement authority grant"This is correct as a reading of the contract, and I made the same argument myself last round. It is nonetheless overridden by an explicit maintainer decision: he rejected the narrowing outright in an interactive session — "That was wrong. You need symmetric authority." That is the contract's designated channel for direction, and it is his system. So symmetric authority is restored, and With the approval gate gone, the loosening evidence bar is now the control that stands in its place, so it is stated as load-bearing rather than procedural: evidence the guard fired on correct mandated work, shipped alone, with the report naming what protection was removed and what now covers that risk. AcceptedP1 — raw error transcripts in /tmp. Correct and my mistake: the output-boundary wrap only covers stdout, so the scratch file leaked around it. Now P2 — timeouts from prose. The third appearance of the prose-fabrication class, after denials and sleeps. Now read from real tool results. The real-corpus count fell 35 → 8, so the inflation was material, not theoretical. P2 — generic secret assignments. Detector/redactor parity broke a second time. Rather than patch it again, the suite now asserts every credential shape is both detected and redacted, so adding a shape to one list fails the build. P2 — Codex P2 — Codex in collision metrics. Self-defeating for a cross-instance metric: the sibling half of the interaction was invisible. P2 — near-miss false positives. Flagging every P2 — 30-PR cap. Saturates on a weekly window and reads as a real plateau. Raised to 300. P2 — product-repo ban contradiction. Scoped to product work; a submodule's NoteThe first push of this commit was blocked by GitHub secret scanning — my test fixtures contained credential-shaped literals. That is push protection working correctly, and it was right to stop me. Samples are now assembled at run time from fragments and substituted into fixtures, so no credential-shaped literal exists on disk while the detector still sees complete, well-formed tokens. Tests are unchanged in strength: 48 passing. |
…nding'
Three review rounds in one session each caught the same shape: the underlying
finding was correct, and the DEFECT was in widening it into a rule covering a
scope that was never tested.
verified: this call site leaks → rule: 'redaction is at the output
boundary' → other call sites still leaked
verified: set -- $x breaks in zsh → rule: written into the cross-tool
contract → bash agents told working code was broken
and handed a syntax error
verified: read <<< works bash+zsh → rule: 'use when shell is unknown'
→ syntax error under POSIX /bin/sh
The widening is invisible precisely because the finding under it is sound, so
it survives self-review and is only caught by someone testing the edge. Adds
the check (name the scope, test one case from each part of it) and the
corollary for fixes: close the class and add a guard that fails on the next
member, rather than trusting review to catch it.
Evidence is this session's own review history — the improver's own subject
matter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1aa0f346fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Seven review findings, one of them constitutional. PROFESSIONAL-WORK BOUNDARY. The miner enumerated the ENTIRE host session stores — 614 project directories, 67 of them outside the portfolio — which can include employer/client repositories the contract places categorically out of scope. Reading such a transcript is itself an interaction with excluded material, so scope is now enforced BEFORE anything is read, and FAILS CLOSED: a session whose scope cannot be established is excluded, and an explicitly empty allowlist refuses to scan rather than defaulting to everything. Scoping revealed a second defect while fixing the first: real Codex sessions record a cwd under $CODEX_HOME/worktrees/<id>/monorepo, NOT under $MONOREPO, so a naive allowlist would have silently excluded the entire Codex instance while the scorecard still claimed to cover both agents. Matched narrowly (the worktree's basename must equal the portfolio root's) rather than allowlisting all of the worktrees directory. Outcomes were measured on the monorepo alone, but both agents ship most of their work in the product repos: the real portfolio count for a 3-day window is 373 merged PRs against the 49 previously reported. Repo list is derived from .gitmodules so it follows the portfolio map instead of going stale. Guard denials read only the Claude corpus and schema, so a guard firing on the Codex sibling reported as "no blocked actions" for the whole deployment — with guard-vs-agent being exactly what those counts decide. Also: PEM redaction masked only the header and left the base64 body; the generic-secret detector required 12 chars while the redactor masked from 8, so short assignments reported clean (parity, a fourth time — now covered by the per-shape test); the untrusted-build detector missed the common `git fetch origin pull/N/head:...` form. The sibling-guard rule still said "never edit the other instance's guards at all", contradicting the restored symmetric grant. Reframed as timing rather than permission: apply between ticks, back up first, hand over when timing cannot be controlled, and always report it since the sibling cannot see this memory. 4 more guards (52 total), including the boundary itself and a fail-closed check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All review findings addressed in @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c901059ade
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…trics Seven review findings. Two were boundary HOLES I introduced myself while fixing the boundary — both from relaxing a check to make tests pass. Anchoring. The path matcher was an unanchored substring test, so a DIFFERENT repository whose name merely extends an allowlisted one was admitted: the slug for /Users/x/git-personal/monorepo-client contains the slug for .../monorepo — exactly the professional-repo-next-door case the boundary exists to stop. Now anchored at a component boundary; the only legitimate continuation of a project slug is Claude's `--worktrees` marker, since a single dash starts a different repository name. Codex worktree identity. Matching on basename alone accepted any repo checked out as $CODEX_HOME/worktrees/<id>/monorepo, including a client repo with the same name. Identity is now confirmed by the worktree's origin remote, failing closed when it cannot be read. Reading before scope was known. The cwd probe read the first 64 KiB, which spills past session_meta into real transcript content — so an out-of-scope transcript was partially read by the very check meant to prevent reading it. Now reads two lines. Guard denials lost their is_error filter when I consolidated onto tool_result_text, so a SUCCESSFUL output beginning "Blocked:" — an application log, a test fixture — counted as a guard firing. These counts decide guard-vs-agent, so inflating them argues for loosening a guard that never fired. Timeout victims listed every command description in the corpus, so a window with zero timeouts still produced a confident-looking list. Now correlated by tool_use_id to results that actually timed out. Also: generic-secret values rejected punctuation (`token=abc:def…` read clean), and sleeps with non-literal delays (`sleep "$d"`) counted as zero while being exactly the busy-wait the metric exists to catch. 7 more guards (59 total), including the monorepo-client case by name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 4 — all 7 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ba4e6ce339
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Seven findings. The headline is that my anchoring fix was still a hole. Allowing a bare `--` continuation of a project slug still admitted a DIFFERENT repository: `monorepo--client` slugs to `…-monorepo--client` and matched. That is the third iteration of one hole — unanchored substring, then `--`-anything, now restricted to the two markers that actually exist (`--claude-worktrees-` and `--git-modules-`, both verified against the live store). Each iteration looked like a fix and left the same class open, which is why the guard now names `monorepo--client` explicitly. Codex origin validation had the same shape: `*github.com/org/*` also matches `https://notgithub.com/org/…` and internal mirrors that merely contain the string. Now matched against exact remote forms, host anchored. Failure metrics read every tool result regardless of is_error, so a SUCCESSFUL command printing an old CI log containing "Command timed out after" or "non-fast-forward" counted as a fresh timeout, race, or collision. Split out a failure-only helper; every "something went wrong" metric uses it. Detector/redactor parity broke a FIFTH time — redact() treats `;` as part of a secret value while the detector excluded it, so `password=abc;def…` was masked on stdout and reported clean by the leak scan. The A2A section counted every file under sessions/ with a raw find, bypassing the scope filter entirely — so professional sessions the corpus deliberately excluded were still counted in the cross-instance scorecard. Reverts were still monorepo-only while merges went portfolio-wide, so a reverted submodule change reported "nothing needed reverting". Test runners (`go test`, `dotnet test`, `pytest`, `cargo test`) execute a checked-out ref's code exactly as builds do and were missing from the untrusted-code candidates. 6 more guards (65 total). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 5 — all 7 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51ac9217c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Round 6 — all 4 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 045769e594
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…n gaps Seven findings. Codex failures were reporting ZERO. Verified against live sessions: real Codex output records carry keys type/id/call_id/output and NO is_error or status field, so the errored-only filter from the previous round dropped every Codex failure — and zero failures reads as a healthy instance. Now honours a flag when present and otherwise falls back to explicit failure markers within real tool OUTPUT records. Weaker than the structural Claude flag, and labelled as such, but a stated approximation beats a silent zero. PEM redaction was line-based, so a multi-line key block had only its BEGIN line masked while the base64 body and END line were emitted. Now masks both markers and standalone key-material lines. The credential detector ran case-sensitively while redact() is case-insensitive, so `TOKEN=` / `API_KEY=` were masked on stdout yet reported clean by the leak scan — parity breaking a SIXTH time, in a sixth distinct way. Argument-parse errors print before `main | redact` is installed and echoed the offending value, which can itself be a credential. They now name the option only. Outcome queries coerced any gh failure to 0, so an unauthenticated or rate-limited run reported "nothing merged, nothing reverted" instead of a failure. Failed repos are now named and the totals carry an explicit INCOMPLETE warning. Codex scope probing read two lines; one suffices for the metadata record. Test assertions still used GNU `\s` in three places, which the macos-latest CI leg would fail. Two self-inflicted bugs found while fixing the above, both caught by the suite rather than by review: an apostrophe in a comment inside a single-quoted jq program terminated the string, and a new `--section` validator rejected `a2a` because it contains a digit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 7 — all 7 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef69f181ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ing signal Four findings, all of the "silent zero / silent no-op" family this PR keeps circling. Codex denials and collisions were reporting ZERO. The guard-denial detector still required an is_error/status field that Codex records do not have — I had fixed that in the failure-text helper last round and missed the denial path, so the safety scorecard reported no Codex guard firings and the A2A section no Codex races. The denial SHAPE is itself the discriminator there (bare output never matches `Blocked:`/`Permission to use`), so the flag is not needed. A misspelled `--section` passed validation, made every `want` test false, and exited 0 after printing just the banner — a scheduled section run looking successful while producing no metrics. Now validated against the real names. PEM redaction left short body lines visible. My first fix for that masked any base64-looking line, and against the real corpus it collapsed 30 distinct guard-denial entries into one unreadable bucket — destroying real signal to catch a rare shape, which is its own kind of failure. Replaced with a sed RANGE between the BEGIN/END markers: stateful across lines, precise, and verified to leave the denial list readable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 8 — all fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29e21de2b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…op replays Round 9. Three findings fixed; three refuted with evidence because they ask for mutually incompatible things. Fixed. A repo whose name IMITATES Claude's worktree marker (`monorepo--claude-worktrees-client`) was admitted. The real marker has a fixed shape — adjective-name-hex6, verified against the live store — so it is now matched exactly. Six older fixtures used made-up suffixes (`z9`, `a1`, `abc123`) and correctly began failing: the same fixtures-must-match-reality lesson as the invented Codex schema. Fixed. Quoted secrets are stored in JSONL with ESCAPED quotes, so a raw grep saw a backslash where the value begins and missed them, while redact() — which runs on decoded output — masked them. The scan now reads decoded strings as well as raw text. Fixed. Codex failure markers are now anchored to the START of output. A real failure leads with its marker; a replayed log mentions one mid-stream. This is the middle path between the previous two rounds, which asked first for a flag that Codex records do not have (reporting ZERO failures) and then for no flag at all (counting replays). Refuted, with the reasoning recorded on the PR: requests to determine scope without reading anything are circular — an `ls` of directory names and one metadata record are the minimum possible, and reading nothing means excluding everything. Verified across live sessions that `session_meta` is always the first record, so one line is both sufficient and minimal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 9 — 3 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4916e7e064
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…p instead Round 10. Six findings, and three of them were the same oscillation for the fourth time — so this commit ends it with evidence rather than another tuning pass. Rounds 7-9 alternated between requiring a Codex error flag (which reported ZERO failures) and matching output text (which counted replayed logs), then anchoring that text to the start (which then missed a real push rejection behind its banner). All three were tuning against a format I had never observed. Checked it properly: across 40 live Codex sessions there are ZERO harness-style failure or denial markers, output records carry no is_error/status, and that instance runs approval-policy=never so it is not denied by design. There was nothing to detect. The fallback is removed, and the report now states plainly that denial detection is Claude-schema only and that a zero says nothing about Codex — an unmeasured surface, not a clean one. The test asserts the DISCLOSURE, which is the guarantee that can honestly be made. Also fixed, all real: - A slurped `jq -rs` failed the WHOLE file on one malformed record, silently dropping every valid record with it. Now tolerates unparseable lines. - `xargs grep` split session paths containing spaces, so hook decisions reported "(none recorded)" on any host with a space in its home path. - Heredoc bodies were counted as sleep commands: a command writing a fixture or doc containing `sleep 60` inflated the busy-wait metric — including this suite's own fixtures. - Guard denials now match harness-emitted shapes (`<tool_use_error> Blocked:`, the full `Permission to use <Tool> with command` phrasing) rather than any stdout beginning "Blocked:", which an application or firewall log produces. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 10 — all 6 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9aa8ff9d7b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…efects Round 11. Thirteen findings; these six were genuine defects, several repeats of classes I had already claimed closed. The boundary hole again, fourth iteration and my own asymmetry: I tightened the `--claude-worktrees-` marker to Claude's real slug format last round and left `--git-modules-[a-z0-9-]+` wide open, so a neighbouring `monorepo--git-modules- client` still matched. Both markers now resolve against reality — the submodule alternation is derived from .gitmodules, the same source as the scope paths, and falls back to matching NOTHING when no submodules are readable. The signal trap removed the scratch file but never exited, so a HUP/INT/TERM from the scheduler cleaned up and then kept running. Credentials were counted TWICE — the decoded-plus-raw scan added last round emits the same match from both streams, doubling every count in the leak table. `echo sleep 60` counted as a busy-wait: whitespace was accepted as a command boundary, so a sleep mentioned as an ARGUMENT scored the same as one executed. Codex command literals kept their escaped `\n`, so a multiline command matched only on its first line. A non-slurped `jq` still aborted at the first malformed record, dropping every valid record after it. Test fixtures: the .gitmodules fixture used `printf '%s'` with `\t`, writing a literal backslash-t that git config cannot parse — the test failed for a reason unrelated to the code it was checking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 11 — 6 real defects fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 904274030b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… matcher Round 12. Six findings; two were direct consequences of my own last two rounds, which is why one surface is now removed rather than tuned again. Removed, not fixed: Codex denial detection. Five rounds alternated between requiring an error flag (counting zero) and matching output text (counting a `cat` of an old log). That instance has no error flag, showed no denial text across 40 live sessions, and runs approval-policy=never. A matcher for a shape never observed cannot be made correct by tuning, so the surface is gone and the output states the gap — the same decision already taken for Codex failures. Real gap against my own spec: the skill's scorecard requires injection_attempts_in_corpus and the safety section never emitted it. Now surfaced — and it immediately found 22 hits on the live corpus, which turned out to be the agent definition and this skill quoting those phrases AS EXAMPLES. The detector cannot distinguish an attack from documentation about attacks, so that limitation is stated in the output: check a hit came from an issue/PR/CI body, not from the definition text, before treating it as real. My previous fix requiring `sleep` at a command position then missed the most common real busy-waits — `while ...; do sleep 60; done` and `( sleep 20 )`. Both now count; `echo sleep 60` still does not. Interrupts were grepped raw, so prose quoting `"interrupted":true` fabricated them; now read from the structured flag. The credential scan still aborted at the first malformed JSONL record, missing every credential after it. The window selects FILES by mtime, so a resumed older session counts in full. Disclosed as a bound rather than left to imply exactness. 82 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 12 — fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a913a4251a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if [ "$SF_COUNT" -eq 0 ]; then | ||
| echo " (no sessions in window)" |
There was a problem hiding this comment.
When the window contains Codex sessions but no Claude sessions, this branch prints only no sessions in window and skips the note that Codex reliability is unmeasured. This is especially misleading for --section reliability, where no later section exposes CX_COUNT, so a live sibling corpus is reported as absent rather than as a known measurement gap.
Useful? React with 👍 / 👎.
| # Did the work actually hold? Reverts and post-merge red are the quality signal | ||
| # that no amount of green pre-merge CI can substitute for. |
There was a problem hiding this comment.
The skill's required quality scorecard includes post_merge_red, and this section explicitly claims post-merge red is a quality signal, but the implementation only queries merged-PR and revert counts. When a merge leaves main CI red without being reverted, the report contains no signal for that regression and can even conclude that nothing needed reverting, preventing the improver from prioritizing the breakage.
Useful? React with 👍 / 👎.
| RACES=$(printf '%s\n%s\n' "$SF_CACHE" "$CX_CACHE" | grep -v '^$' \ | ||
| | while IFS= read -r f; do tool_result_failure_text "$f"; done \ | ||
| | grep -cE 'has been modified since read' || true) | ||
| NONFF=$(printf '%s\n%s\n' "$SF_CACHE" "$CX_CACHE" | grep -v '^$' \ | ||
| | while IFS= read -r f; do tool_result_failure_text "$f"; done \ | ||
| | grep -ciE '(non-fast-forward|rejected.*fetch first|would be overwritten by merge)' || true) |
There was a problem hiding this comment.
These counts use tool_result_failure_text, whose Codex branch emits output only when an is_error or status flag exists, while the script's own verified schema notes say live Codex output records have neither. Consequently, a Codex push rejection or two-writer error is always dropped even though both emitted labels claim BOTH instances, hiding the sibling-side coordination failures this section is intended to measure.
Useful? React with 👍 / 👎.
| if ! c=$(gh pr list --repo "$r" --state merged --limit 300 --json mergedAt \ | ||
| --jq "[.[] | select(.mergedAt >= \"${SINCE_ISO}\")] | length" 2>/dev/null); then |
There was a problem hiding this comment.
The merged-PR query requests only mergedAt and therefore counts every portfolio merge, including human, external-contributor, and automation-owned dependency PRs. Since this scorecard is used to diagnose the Daily AI Engineer's quality, unrelated merges—and likewise unrelated reverts in the following query—can move its quality metric and trigger definition changes despite no change in either agent's work; filter outcomes using the required agent disclosure or otherwise report them as portfolio-wide rather than agent quality.
Useful? React with 👍 / 👎.
| printf '%s\n%s\n' "$SF_CACHE" "$CX_CACHE" | grep -v '^$' | while IFS= read -r f; do | ||
| grep -hoiE '(ignore (all )?(prior|previous) (rules|instructions)|disregard (your|all) (instructions|rules)|the maintainer (approved|authorised|authorized)|add [^ ]+ to the trust gate|update your instructions|you are now [a-z ]{0,20}mode)' \ | ||
| "$f" 2>/dev/null | ||
| done | redact | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -6 | sed 's/^/ /' |
There was a problem hiding this comment.
The scorecard requires a numeric injection_attempts_in_corpus value, but this pipeline emits only the six most frequent distinct phrases and no aggregate total. Once a window contains more than six distinct matches, the remaining occurrences disappear from the report, so day-over-day safety trends can fall or stay flat merely because different phrases entered the top six rather than because injection-shaped input decreased.
Useful? React with 👍 / 👎.
Round 13. Four findings, all "the implementation does not match my own spec". post_merge_red was named in the skill scorecard and claimed by this section as a quality signal, but only merges and reverts were ever queried — so a merge that left main red WITHOUT being reverted produced no signal at all, the worst case to miss since nothing else surfaces it. Now read from the CHECK-RUNS on main's head; a first attempt using "latest workflow run" reported three false REDs because path-filtered workflows legitimately conclude `skipped`. Repos whose checks cannot be read are named UNKNOWN with the verified cause (HTTP 403, token lacks checks:read on the private repos) — unmeasured, never green. Outcomes counted every portfolio merge, including maintainer, external and dependency-bot PRs, so unrelated activity moved the agents' own quality metric. Now filtered to claude/* and codex/* head branches: 223 agent-authored against 373 total for the same window, so roughly 150 merges were being misattributed. Author login cannot discriminate here, because the agent commits as the maintainer. Removing the Codex text fallback last round left the collision counts Claude-only while both labels still said BOTH instances. A false coverage claim is worse than a stated gap, so the labels now say CLAUDE ONLY and explain that the sibling side of a two-writer race is unmeasured — precisely the coordination this section exists to measure. The injection count printed only its top six phrases with no aggregate, so a day-over-day trend could fall merely because different phrases entered the top six. Now reports total occurrences and distinct phrases alongside. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
64d46c5 shipped a syntax error: an added `echo` line lost its closing quote, so the script would not parse at all. I chained the verification with `;` rather than `&&`, so the failing `bash -n` did not stop the commit — the exact "success reported, nothing checked" shape this PR has been cataloguing, applied to my own release step. Verified now with fail-fast gating: parses, 82 tests pass, and the outcomes section runs against the live portfolio. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Round 13 — 4 fixed in
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58645969c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case "$cwd" in | ||
| "$CODEX_HOME"/worktrees/*/"$(basename "$p")"|"$CODEX_HOME"/worktrees/*/"$(basename "$p")"/*) | ||
| [ -d "$cwd" ] || continue | ||
| url=$(git -C "$cwd" remote get-url origin 2>/dev/null) || continue |
There was a problem hiding this comment.
When an out-of-scope checkout exists at $CODEX_HOME/worktrees/<id>/monorepo (the collision scenario described immediately above), this command reads that repository's Git configuration before deciding to exclude it. That directly violates AGENTS.md's hard exclusion to “Never discover, enumerate, search, inspect metadata/content/CI … or otherwise interact with such repositories.” Validate the candidate against worktree information obtained from the already-authorized portfolio repository rather than running Git inside an untrusted candidate path.
Useful? React with 👍 / 👎.
| if ! rcraw=$(gh api "repos/$r/commits" -X GET -f since="${SINCE_ISO}T00:00:00Z" --paginate \ | ||
| --jq '[.[]|select(.commit.message|test("^Revert";"i"))]|length' 2>/dev/null); then |
There was a problem hiding this comment.
When a maintainer, dependency bot, or external contributor's change is reverted during the window, this query increments RTOTAL solely because the commit subject starts with Revert. The merge metric above deliberately filters to claude/* and codex/*, but this other quality metric remains portfolio-wide, so unrelated activity can still worsen the agents' scorecard and drive an unjustified definition change. Correlate each revert with the reverted agent-authored PR/commit, or explicitly keep it outside the agent-quality metric.
Useful? React with 👍 / 👎.
| SINCE_ISO=$(date -u -v-"${SINCE_DAYS}"d '+%Y-%m-%d' 2>/dev/null \ | ||
| || date -u -d "${SINCE_DAYS} days ago" '+%Y-%m-%d' 2>/dev/null) |
There was a problem hiding this comment.
Converting the cutoff to a date discards the current UTC time: for example, --since-days 1 at 23:00 queries merges and reverts from midnight of the previous day, yielding an approximately 47-hour outcome window while the transcript metrics cover roughly 24 hours. This makes the scorecard's declared window_days inaccurate and distorts baselines used to verify improvements. Keep a full UTC timestamp and use it for both mergedAt filtering and the commits API.
Useful? React with 👍 / 👎.
| -e 's/-----(BEGIN|END) [A-Z ]*PRIVATE KEY-----/<redacted-private-key>/g' \ | ||
| -e 's/(-----BEGIN [A-Z ]*PRIVATE KEY-----)[^-]*/\1<redacted-key-material>/g' \ | ||
| -e 's/(eyJ[A-Za-z0-9_-]{6})[A-Za-z0-9_.-]{20,}/\1…<redacted-jwt>/g' \ | ||
| -e 's/((secret|token|password|passwd|api[_-]?key)["'"'"']?[[:space:]]*[:=][[:space:]]*["'"'"']?)[^"'"'"'[:space:],}]{8,}/\1<redacted>/gI' |
There was a problem hiding this comment.
When a quoted generic credential contains whitespace, such as password="correct horse battery staple", this expression does not match it because the value class stops at the first space; the reliability report consequently prints the full password from an errored tool result. This violates the agent's non-negotiable requirement that credential-shaped strings be redacted rather than echoed. Handle quoted values through their matching closing quote, while retaining the existing delimiter-based handling for unquoted assignments.
Useful? React with 👍 / 👎.
| - Metric fell → close the hypothesis, keep the change. | ||
| - Metric unchanged → the diagnosis was wrong. **Say so**, revert or reshape. Do not layer a second | ||
| guess on top of an unverified first. | ||
| - Metric rose → **revert first, diagnose after.** |
There was a problem hiding this comment.
When the next window contains more sessions or a different workload mix, a raw failure count can rise even if the change reduced the failure rate; the telemetry banner itself says these counts are directional rather than exact. This unconditional instruction would then revert a correct improvement without establishing causality. Compare normalized rates over equivalent volume, or require repeated comparable windows and a verified regression before reverting.
Useful? React with 👍 / 👎.
| echo " [BOTH instances: ${SF_COUNT} Claude + ${CX_COUNT} Codex sessions]" | ||
| echo " bash timeouts .............. ${TIMEOUTS} (each = a foreground block that produced nothing)" |
There was a problem hiding this comment.
When a Codex command times out, its observed output record has no is_error or status flag, so tool_result_failure_text deliberately contributes no Codex text and TIMEOUTS cannot count the event. Nevertheless this section labels the metrics as covering both instances and can report zero without any gap warning, making a Codex-only timeout window look healthy. Label the timeout metric Claude-only or report the Codex surface as unmeasured, as the reliability and A2A sections already do.
Useful? React with 👍 / 👎.
| fi | ||
| fi | ||
| done | ||
| echo " (no output above = loaders agree with the constitution)" |
There was a problem hiding this comment.
When either loader drifts in any way other than the two hard-coded retired-promotion phrases, this loop emits nothing and the following message declares that the loaders agree with the constitution. For example, a loader that reintroduces a different retired permission rule receives a clean result even though this section is presented as the loader-to-constitution drift check. Narrow the message to the specific signatures tested, or implement a general comparison before claiming agreement.
Useful? React with 👍 / 👎.
…-ai-agent-15c44c # Conflicts: # .github/workflows/ci.yaml
Codex reviewed Main had moved four commits in the meantime, so 82 tests pass and the committed script parses — gated with Re-requesting review at the merge head, since any push stales the previous green. @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dee4e49aab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Self-gate on the workflow too, so a change that breaks this | ||
| # job's own wiring still runs the test it gates. | ||
| - '.github/workflows/ci.yaml' |
There was a problem hiding this comment.
Adding .github/workflows/ci.yaml only under agent-telemetry removes it from the preceding active-projects filter, where the parent commit used it to self-gate drift-check-active-projects. Consequently, a workflow-only change can now alter or break that job's wiring while the job is skipped; retain this path under active-projects as well as adding it here.
Useful? React with 👍 / 👎.
| git -C "$MONOREPO" config --file .gitmodules --get-regexp '\.path$' 2>/dev/null \ | ||
| | awk '{print $2}' | while IFS= read -r sub; do | ||
| [ -n "$sub" ] && printf ':%s/%s' "$MONOREPO" "$sub" | ||
| done |
There was a problem hiding this comment.
Do not automatically allowlist every .gitmodules path: the inspected file includes outside-devantler-tech checkouts such as github/personal/.github-public and github/personal/profile, so any Claude transcript under those paths is scanned by an unattended run. This violates the AGENTS.md professional-work hard exclusion, which requires current explicit confirmation before even read-only access to a repository outside devantler-tech; build the default from confirmed portfolio repositories rather than all submodules.
Useful? React with 👍 / 👎.
| cwd=$(head -n 1 "$f" 2>/dev/null \ | ||
| | jq -r 'select(.type=="session_meta")|(.payload.cwd? // .cwd? // empty)' 2>/dev/null | head -1) | ||
| if in_scope_cwd "$cwd"; then printf '%s\n' "$f"; fi |
There was a problem hiding this comment.
When the global Codex session store contains a professional/out-of-scope session, this reads its first record before in_scope_cwd can reject it. The hard exclusion in AGENTS.md explicitly forbids inspecting even metadata or content from such repositories, so limiting the read to one metadata line does not fail closed; use a separately scoped index/store or omit Codex sessions whose scope cannot be established without opening them.
Useful? React with 👍 / 👎.
| sqlite3 -readonly "$CODEX_HOME/logs_2.sqlite" \ | ||
| "SELECT ' '||level||': '||COUNT(*) FROM logs WHERE ts > ${CUT} GROUP BY level ORDER BY COUNT(*) DESC LIMIT 6;" 2>/dev/null \ |
There was a problem hiding this comment.
When logs_2.sqlite contains activity from any professional/out-of-scope project during the window, this query reads and aggregates those records because it has no portfolio predicate or correlation to CX_CACHE. That directly contradicts the AGENTS.md hard exclusion against inspecting metadata from professional repositories; omit this metric unless it can be restricted to already-verified in-scope sessions.
Useful? React with 👍 / 👎.
| NONFF=$(printf '%s\n%s\n' "$SF_CACHE" "$CX_CACHE" | grep -v '^$' \ | ||
| | while IFS= read -r f; do tool_result_failure_text "$f"; done \ | ||
| | grep -ciE '(non-fast-forward|rejected.*fetch first|would be overwritten by merge)' || true) |
There was a problem hiding this comment.
The phrase would be overwritten by merge is emitted when a pull, checkout, or merge is blocked by the current worktree's own uncommitted changes; it does not establish that another instance collided with this run. Counting it in NONFF therefore turns ordinary local-worktree hygiene failures into cross-instance coordination evidence, which can drive the improver toward an unrelated claim-protocol change; classify this signature separately from actual non-fast-forward push failures.
Useful? React with 👍 / 👎.
| echo " [BOTH instances: ${SF_COUNT} Claude + ${CX_COUNT} Codex sessions]" | ||
| echo " bash timeouts .............. ${TIMEOUTS} (each = a foreground block that produced nothing)" |
There was a problem hiding this comment.
For real Codex records without an is_error or error status flag, tool_result_failure_text deliberately emits nothing, so TIMEOUTS cannot include Codex timeouts; nevertheless this output labels the following metrics as covering both instances. In a Codex-only timeout window it will confidently print zero under the BOTH instances banner, hiding the exact efficiency regression the improver is meant to detect; label the timeout metric Claude-only and disclose the Codex gap.
Useful? React with 👍 / 👎.
| if [ "$SF_COUNT" -eq 0 ]; then | ||
| echo " (no sessions in window)" |
There was a problem hiding this comment.
When the selected window contains Codex sessions but no Claude sessions, this branch prints (no sessions in window) and skips the later note that explains Codex reliability is unmeasured. That reports an empty corpus rather than a known schema gap and can make a Codex-only period look healthy; say that there are no Claude sessions and still report CX_COUNT as unmeasured.
Useful? React with 👍 / 👎.
1 participant
Why
The Daily AI Engineer improves the products, but nothing improves the Engineer itself from the outside. The existing self-improvement skill is a single run reflecting on its own memory — so it cannot see what only shows up across hundreds of runs or between the two sibling instances: recurring failures, silent waste, and drift between what the loaders say and what the constitution says.
That gap is real, not theoretical. Running the new telemetry against the live corpus surfaced four defects on the first pass — including both loaders still instructing the agents to follow a rule retired in #2239.
What
This PR carries the deployment-specific half: the telemetry miner that reads this deployment's two instances (Claude Code + Codex session stores, both memory stores, both loaders, GitHub outcomes) and its tests, wired into CI on both OSes.
The generic, redistributable half ships in the shared libraries instead of here, per the portability principle:
agent-improvementskill → feat(agent-improvement): add the agent-improvement skill agent-skills#71agent-improveragent, bundled into theautomated-ai-engineerplugin → feat(automated-ai-engineer): add the agent-improver meta-engineer agent agent-plugins#73The agent and skill in
.claude/here are this deployment's instances of those, matching howportfolio-maintenance,product-engineeringandself-improvementalready live in both places.Authority: you granted full symmetric authority over the prose definition, both loaders, and the enforcement layer (session, 2026-07-18). I recommended an asymmetric grant and you overrode it; the definition puts the weight on evidence, reversibility and audit rather than a permission gate.
Operational note: the routine is already live as a scheduled task (daily 06:30, offset from both agents' ticks), and the two loader drifts it found are already fixed in place with
.bakbackups.