feat(improvement): converge on one measured proposal-to-activation path

[drewstone]

Decision

Do not add another all-in-one improve() API yet.

Converge the existing packages and product on one explicit improvement path:

observation -> candidate -> measured comparison -> proposal -> review -> authorized transition -> transition result -> live outcome

Current-task guidance remains a separate, temporary effect.
It may use the same observations and feed later candidate search, but it is not a persistent change and must never be reported as one.

The first implementation is one shared-contract release train followed by deletion of duplicate paths, not another feature layer.

Customer Outcome

A customer connects an agent, its declared profile, and whatever task evidence its environment can provide.

The system then does the strongest honest thing the available evidence permits:

Available evidence	Product output	Persistent change allowed by default
Traces only	Findings, recurring failure patterns, recommended actions, candidate hypotheses, and missing-instrumentation requests	No
Traces plus recorded tool results	Replayed comparisons with divergence and limitations shown	No
Frozen starting state, representative tasks, and executable checks	Paired baseline-versus-candidate measurements with quality, cost, latency, uncertainty, and regressions	Yes, after review

The customer sees what would change, why it was proposed, every measured dimension, uncertainty, cost, latency, limitations, and the exact target state.
They can approve, reject, request changes, give feedback, activate, or return to a prior state.
An approval is not reported as active until the target confirms the exact state transition.

For Uri's current trace-only data, this means a useful report and concrete profile or instrumentation hypotheses are valid outputs.
Claiming measured lift or activating a change is invalid until the Symphony repository/profile and replayable tasks or equivalent checks are connected.

Audiences

Audience	Contract they need
Primary: application developer	One documented path from a run to a reviewable proposal and an explicit deployment adapter, with no Intelligence product dependency
Customer operator and reviewer	A complete decision view, feedback, approval policy, exact activation status, and return to a prior state
Running agent or driver	Bounded current-task guidance and a way to submit candidate changes without gaining activation authority
Proposer and optimizer author	A registry contract for applicability, inputs, outputs, cost, lineage, and evaluation
Agent-system adapter author	Lossless session intake plus deterministic profile and candidate materialization for their system
Hosted product owner	Tenant storage, billing, policy, delivery, concurrency, and deployment adapters around the shared documents
Security, compliance, and research users	Immutable provenance, exact state, complete measurements, explicit limitations, and exportable run artifacts

Fresh Audit: July 17, 2026

The core system is substantially built, but its published releases do not currently compose.

Package	Current published version	Relevant dependency contract
@tangle-network/agent-interface	0.30.0	Canonical candidate, experiment, comparison, proposal, review, and activation documents
@tangle-network/agent-eval	0.122.1	Requires agent-interface ^0.30
@tangle-network/agent-knowledge	3.0.1	Requires agent-eval ^0.122.1 and agent-interface ^0.30
@tangle-network/agent-profile-materialize	0.5.1	Requires agent-interface 0.30
@tangle-network/traces	0.9.0	Depends on agent-eval ^0.118.3, which excludes 0.122.1
@tangle-network/agent-runtime	0.94.13	Requires eval <0.121, interface <0.27, knowledge ^1.12.1, and materializer 0.3.2

Four of the six published packages align on the current contract.
agent-runtime and traces are the release blockers.

Published npm state and local source state also differ.
The current local knowledge checkout still declares version 1.10.0 and older runtime/eval dependencies, while npm contains the newer split candidate/promotion API.
Implementation must begin from freshly fetched default branches and check their packed output against the published tarballs before editing.

The Intelligence product currently requests older runtime/eval/interface releases, forces most interface consumers to 0.27.2, and also loads the workspace materializer against interface 0.30.
A clean product install therefore contains two incompatible interface contracts.
The product also emits schemaVersion: 1 in profile diffs even though the current AgentProfileDiff contract removed that field.

These checks replace the previous assumption that the missing capability was merely composition.

The latest traces@0.9.0 also reproduced an ingestion-quality problem on this work's own Codex session.
It read 110,240 spans from one session, reported five malformed source records, captured no task-quality label and no cost for 1/1 run, and returned zero analyst findings while retaining 51 deterministic signals.
It also marked the run failed even though its cited command output explicitly said Process exited with code 0; the fallback text matcher interpreted error: inside successful stdout as execution failure.
Raw input must survive before parsing, structured status must outrank output text, and reports must expose retained signals and missing coverage even when an analyst emits zero findings.

What Already Exists

Do not rebuild these capabilities in the product or runtime.

Owner	Existing capability to keep
agent-interface	Full agent profiles and diffs; immutable candidate bundles; frozen experiments; paired comparisons; proposal, review, and activation documents; canonical hashes
agent-profile-materialize	Translation from one profile into native Claude Code, Codex, OpenCode, Pi, and other workspace files
agent-eval	Trace analysts; code-session intake; GEPA, FAPO, SkillOpt, ACE, policy, memory, composite, evolutionary, HALO, coding, research, and parameter-search proposers; uncertainty; proposer comparison; cross-surface interaction analysis
agent-knowledge	Research, retrieval, answer quality, wiki, citation, freshness, memory adapters, candidate-copy evaluation, and knowledge promotion
agent-runtime	Agent execution, temporary analyst guidance, candidate generation, optimization jobs, exact execution work in PR #547, and knowledge worker orchestration
Intelligence product	Tenant storage, trace ingestion, reports, consultant output, datasets, optimization UI, candidate review, approval quorum, rollback history, durable delivery, Slack, webhook, GitHub, Linear, and email integrations

The Intelligence approval route already records an internal artifact version and moves an active pointer atomically.
It does not yet bind that action to the shared measured proposal, shared review, exact candidate digest, expected current digest, or a portable activation result.

The One Model

1. Observe

Retain the raw run events before analysis.
Normalize Claude Code, Codex, OpenCode, Pi, and other supported sessions through the existing eval and traces adapters.
Preserve unknown events instead of dropping them.
Persist or quarantine the original tenant-owned bytes before parsing so malformed records remain recoverable without placing sensitive content in reports.

2. Guide the current run when useful

Existing runtime supervision may send temporary guidance to the current worker or its driver.
Guidance must state its source, recipient, and lifetime, and it must be retained as an observation.
It does not create or activate a persistent candidate.

3. Search for candidates

Candidate sources include existing artifacts, open-source skills/tools/MCP servers/extensions, tenant-authored changes, in-box drift, and generative proposers.

Each proposer advertises the inputs it needs, the changes it can produce, expected cost, and safety requirements.
The planner runs every applicable proposer that fits the declared budget and records every considered, selected, skipped, failed, and rejected proposer with a reason.
It does not blindly run every installed proposer.
An explicit diagnostic mode may do that for comparison.

External research is a candidate source, not a special promotion path.
Discovered artifacts must be pinned, licensed, security-reviewed at the level required by policy, materialized, and measured like generated artifacts.

4. Freeze the exact candidate

One AgentCandidateBundle is the complete executable candidate.
It contains the full candidate profile, pinned resources, code state, execution definition, knowledge state where applicable, and evaluation memory isolation.

AgentProfileDiff is the editable explanation of a profile change.
AgentCandidateBundle is the frozen full state that can actually run.
The product's generic artifact row is only a tenant-owned storage envelope for these shared documents.
These three roles must not grow competing schemas.

The current contract has two inconsistencies to resolve before adoption:

1. AgentImprovementSurface manually lists ten labels while AgentProfileDiff covers fifteen profile axes and nested skill resources.
   Changed surfaces must be derived from the baseline and exact candidate rather than supplied as a second source of truth.
2. The candidate's memory field describes evaluation isolation, not a durable memory improvement.
   Rename that concept and either represent durable memory through the knowledge candidate contract or add one immutable memory candidate reference.
   Do not let one field mean both.

Profile, code, knowledge, and durable memory remain specialized data because they have different materialization and promotion semantics.
They share one candidate envelope and one comparison protocol; they do not need one vague universal artifact type.

5. Measure before proposing

Run baseline and candidate on the same representative tasks, budgets, model snapshots, starting state, and checks.
The tasks used for final comparison must not be visible to candidate generation.

Use agent-eval's existing paired comparison and cross-surface interaction analysis.
That analysis already retains the full task-by-candidate matrix and compares the best individual change, a naive stack, and an interaction-aware bundle.
This replaces the unique useful behavior claimed by runtime's separate lifecycle implementation.

The comparison must retain all task-level outcomes and report quality, cost, latency, token use, regressions, uncertainty intervals, sample count, termination, and provenance.
If representative tasks or executable checks are missing, return insufficient-evidence rather than manufacturing a promotion decision.

6. Propose

A proposal binds one exact comparison and candidate.
Its customer presentation includes the exact changes, rationale, measured benefits, regressions, cost, latency, uncertainty, limitations, and requested action.
Product-specific consultant prose is a view over this evidence, not a replacement for it.
Do not copy the candidate into another proposal field when the bound experiment and comparison already identify it.

7. Review

The shared review document records approve, reject, or request-changes against one proposal digest.
Product-owned policy may add quorum, roles, funding, and compliance checks.
Review feedback becomes input and lineage for the next search instead of disappearing into UI state.

8. Activate exactly once

Search, analysis, proposal creation, and review never mutate active behavior.

Creating a pull request is a reviewed delivery action, not activation.
It may publish the exact candidate for a customer to merge, but the system reports active only after a target adapter observes the merged or deployed state.

Activation authorizes one exact transition from an expected current state to the measured candidate state.
It requires an idempotency key, expiry, and compare-and-swap semantics.
A retry either returns the original result or already-applied; a changed base returns conflict.

The activation target is one deployable agent-state pointer, not one changed surface.
It contains { targetIdentity, expectedActiveDigest, requestedActiveDigest }.
The requested digest identifies an immutable manifest that may reference profile, code, knowledge, memory, and materialization receipts.
Changed surfaces belong on the comparison and proposal only.

Prefer one atomic deployment pointer whose immutable manifest references the profile, code, knowledge, and memory snapshots.
An adapter that cannot activate a compound candidate as one exact state must return unsupported or split and remeasure the candidate.
It must not partially apply a compound candidate and call it successful.

The current shared activation document permits several targets but defines no atomicity, expiry, idempotency, result, or failure contract.
Replace that ambiguity with one exact target transition and add AgentImprovementActivationResult containing the prior state, requested state, active state, timestamps, outcome, and typed error.

Returning to a prior approved state uses the same transition protocol in reverse.
Do not build a second rollback subsystem.
Adapt the existing product rollback route to the same expected-current digest, idempotency, expiry, audit event, result, and delivery behavior, then delete any rollback path that cannot be expressed as that transition.

9. Measure the live outcome

Later runs identify the active candidate digest.
The product compares live quality, cost, latency, failures, and drift against the measured expectation and can propose reversal when the result degrades.
Observed live association remains distinct from a controlled comparison.

Package Ownership

Package	Owns	Must not own
agent-interface	Portable documents, strict schemas, canonical hashing, exact transition and result contracts	Storage, billing, UI, provider execution
agent-eval	Analysis, proposer registry, search, experiments, paired statistics, uncertainty, interaction analysis	Persistent activation, tenant policy
agent-knowledge	Sources, retrieval, memory adapters, knowledge candidates, candidate quality, knowledge promotion mechanics	General agent execution, product approval
agent-profile-materialize	Deterministic profile-to-workspace compilation and receipts	Optimization decisions
agent-runtime	Agent execution, temporary guidance, worker budgets, exact candidate execution, resumable improvement jobs	Product storage, approval UI, direct unreviewed mutation
traces	Lossless session ingestion and analysis entry points	Candidate promotion
Intelligence product	Tenancy, persistence, billing, policy, review UI, notifications, target adapters, active-state pointers	Portable eval statistics and package contracts

Duplication to Remove

Current path	Disposition
Runtime improve()	Keep as bounded candidate search only, rename its result away from shipped, and remove direct writeBack or profile mutation
Runtime src/lifecycle and defineAgent().lifecycles	Preserve tested behavior through eval's comparison and interaction primitives, then delete the 2,467 non-test lines and 1,461 test lines
Runtime runAnalystLoop() direct application	Restrict to temporary guidance or candidate/proposal output
Runtime src/intelligence/improvement-cycle.ts proposal, review, and evidence contracts	Delete and use interface constructors; runtime may execute an authorized bundle but must not define portable documents
Runtime surface adapter write and open-pr modes	Convert to candidate creation; a review-controlled product delivery adapter may open a PR, while verified merge/deployment remains the activation event
Runtime knowledge job's old implicit promotion result	Use current knowledge candidate preparation during search and promoteKnowledgeCandidate only inside an authorized target transition
Runtime-local candidate and knowledge digest conversions	Replace with current interface and knowledge helpers
Eval runImprovementLoop() automatic PR creation	Return the measured candidate/proposal only; move GitHub effects behind explicit review-controlled product delivery
Product runImprovementLoop name	Rename to its actual analysis/proposal responsibility so it cannot be confused with measured search
Product-local portable proposal, review, statistics, and activation shapes	Replace with shared contracts; keep tenant storage and UI read models
Product profile-diff schemaVersion	Delete when adopting the current interface contract
Stale PR #544 and PR #547 branches	Mine the useful exact-execution work into fresh, dependency-ordered changes; do not merge either conflicting branch wholesale

No product data migration is required for runtime's in-memory ProfileArtifact registry because no live product consumer was found.
Removing its pre-1.0 public API still requires release notes and a migration example for library users.

Surface Choice

The system chooses what to improve from evidence, applicability, and budget rather than a hand-maintained list or a single model guess.

Signal	Candidate sources favored
Repeated instruction misunderstanding	Prompt and instruction proposers
Missing repeatable procedure	Existing-skill search, SkillOpt, or a new skill candidate
Tool unavailable or repeatedly misused	Tool, MCP, hook, permission, or tool-description candidates
Delegation or coordination failure	Subagent, role, driver-guidance, or orchestration candidates
Missing/stale evidence	Research, retrieval, wiki, citation, or knowledge candidates
Bad recall or noisy retained state	Memory policy or memory-store candidates
Implementation defect or missing automation	Code or workflow candidates
Several interacting causes	Composite and cross-surface candidate analysis

The proposal records why each surface was selected and why plausible alternatives were rejected.
The registry, not product switch statements, discovers new proposers.

Start Conditions and Delivery

The hosted product may start the same resumable improvement job from a manual request, a schedule, enough repeated evidence, a measured live regression, or an agent-submitted candidate.
These are triggers, not separate improvement implementations.
Each trigger names the tenant, funding owner, budget, allowed targets, and review policy before spending begins.

Proposal-ready and activation-complete events flow through the existing durable delivery system.
Email, Slack, webhooks, GitHub, and Linear are channel adapters over the same event, and dry-run mode records the event without sending it.

Required User Experiences

1. A trace-only customer receives useful findings and an honest list of what cannot yet be tested.
2. A customer links a repository/profile to sessions and sees the exact prompt, skills, tools, MCP servers, hooks, subagents, code, and knowledge state that was observable or missing.
3. A running agent receives bounded temporary guidance without silently changing future runs.
4. Prompt, skill, tool/MCP, code, knowledge, and memory candidates can each be generated or discovered and measured when their required inputs exist.
5. Open-source candidate discovery can search deeply, pin results, reject unsafe or incompatible options, and compare them with generated alternatives.
6. A compound candidate is selected only after interaction analysis shows it beats its components under the same budget.
7. A customer sees every measured dimension, uncertainty, sample count, cost, latency, failures, and limitation before deciding.
8. Approve, reject, request-changes, and free-form feedback all persist and affect later work.
9. Activation is conflict-safe, retry-safe, exact, and visibly distinct from approval.
10. A prior approved state can be restored through the same exact transition mechanism.
11. Existing Slack, webhook, GitHub, Linear, and email delivery can notify on proposal readiness and completed activation without sending in dry-run mode.
12. Later runs show whether the active candidate actually delivered the predicted outcome and whether it drifted.
13. A package-only adopter can supply their own storage, task suite, review authority, and deployment adapter while using the same candidate, comparison, proposal, review, transition, and result documents without the Intelligence product.

The default customer experience exposes only goal, budget, allowed targets, and approval policy.
Advanced controls may set task suites, objective weights, candidate sources, proposer allowlists, model budgets, evidence requirements, delivery channels, and activation mode.

Implementation Order

A. Freeze the shared contract, then publish one compatible package set

• Replace manual changed-surface declarations with values derived from exact baseline/candidate state.
• Separate evaluation memory isolation from durable memory candidates.
• Define one deployable agent-state target with expected and requested active digests.
• Add expiry, idempotency, conflict behavior, activation result, and typed failure outcomes.
• Model return-to-prior-state through the same transition.
• Add canonical hashing, strict parsing, replay, retry, conflict, and tamper tests.
• Update the sole pre-1.0 contract in place; do not add V1/V2 type names, parallel parsers, or compatibility shims, and write one data migration only if persisted rows require it.
• Publish the interface contract first, then publish compatible eval, knowledge, materializer, traces, and runtime releases in dependency order.
• Preserve at least the capabilities present in interface 0.30.0, eval 0.122.1, knowledge 3.0.1, materializer 0.5.1, traces 0.9.0, and runtime 0.94.13 while removing their incompatible ranges.
• Ensure traces and runtime resolve the same eval and interface releases rather than installing older nested copies.
• Remove runtime-local contracts and conversion helpers now owned by those packages.
• Make runtime build, test, and pack from a clean install with no local links or unpublished forks.
• Install the packed packages together in a blank consumer and prove only one agent-interface version resolves.
• Publish runtime only after that consumer proof passes.
• Update the Intelligence product's dependencies, overrides, and lockfile in one change and prove a clean product install, typecheck, and focused Intelligence tests.

No unrelated composition API lands before this release train is green.

B. Collapse runtime behavior

• Make candidate search non-mutating.
• Use eval's existing paired comparison and cross-surface interaction analysis.
• Port the useful exact-candidate execution from PR feat(candidate): measure exact candidates before activation #547 onto current contracts.
• Route knowledge work through current knowledge primitives.
• Keep eval's generic observe/validate/decide/act engine as a low-level control utility, but do not treat its arbitrary act callback as activation authority.
• Delete runtime lifecycle, artifact registry, direct analyst mutation, and duplicate converters after parity tests pass.
• Keep temporary guidance explicit and separate from persistent optimization.

C. Adopt the shared path in Intelligence

• Use current code-session intake and deep analysts for every supported source.
• Persist the shared candidate, experiment, comparison, proposal, review, activation, and activation-result documents.
• Bind the existing approval route and active pointer to exact shared digests and expected current state.
• Add one active agent-state manifest pointer for compound deployment; existing per-artifact pointers may remain inventory references but cannot independently claim a compound candidate is active.
• Express the existing rollback route through the same transition result and outbox behavior.
• Keep the existing durable delivery system and add proposal-ready versus activation-complete event types.
• Render one decision view with exact changes, task-level results, aggregate metrics, uncertainty, cost, latency, regressions, limitations, target, and feedback.
• Materialize and verify real prompt, skill, tool, MCP, hook, and subagent changes rather than metadata-only placeholders.
• Preserve insufficient-evidence as a first-class product result.

D. Prove the whole customer path

• Run a deterministic fixture from ingestion through report, candidate, comparison, review, activation, live outcome, and return to a prior state.
• Run an isolated own-account test using drewstone329@gmail.com or drew@tangle.tools with external delivery and customer billing disabled.
• Run Uri's current data read-only and prove it produces a trace report, candidate hypotheses, observability gaps, and insufficient-evidence wherever exact replay is impossible.
• After Uri links the Symphony repository/profile, run an isolated replay that proves at least one real candidate can be materialized and compared without customer-side actions.
• Capture all process cost, tokens, latency, retries, proposer decisions, and task outcomes.
• Test two concurrent approvals, duplicate delivery, stale base state, interrupted activation, retry, unsupported target, and partial-application refusal.

Completion Criteria

• One compatible published package set installs with one interface contract.
• Interface remains dependency-free from the higher layers; eval, knowledge, and materializer consume it; runtime composes them; knowledge does not depend on runtime.
• No /tmp links, unpublished forks, duplicate portable contracts, or parallel schema generations are required.
• Candidate search has zero persistent mutation paths.
• Every candidate is immutable, content-addressed, fully materializable, and tied to exact execution inputs.
• Baseline and candidate use the same tasks, state, model snapshot, budget, and checks.
• Final comparison tasks are independent from candidate generation.
• Every metric and every task result is retained, including nulls, failures, cost, latency, termination, and uncertainty.
• Every installed proposer is automatically considered when applicable, with selected/skipped/failed/rejected reasons and spend recorded.
• Cross-surface bundles are measured against individual components and naive stacking.
• Trace-only data cannot produce a false measured-lift or activation claim.
• Structured tool status outranks words found inside stdout, and successful output containing error: cannot become a false run failure.
• Malformed and unknown source records remain recoverable from tenant-owned raw storage, with parse receipts linked to the original bytes.
• Zero analyst findings cannot hide deterministic signals, missing task labels, or missing cost coverage.
• Review feedback is consumed by the next candidate search.
• Approval and activation are distinct persisted facts.
• Activation is exact, expiring, idempotent, conflict-safe, and returns a typed result.
• Compound candidates cannot partially activate and report success.
• Returning to a prior state uses the same transition and is proven.
• The product uses shared documents while retaining tenant, billing, policy, UI, and delivery ownership.
• Real materialization covers prompt, skills, tools, MCP, hooks, and subagents for at least Claude Code plus one other supported system.
• Prompt, skill, tool/MCP, code, knowledge, and durable-memory paths each have focused process tests.
• Fixture, own-account isolated, and Uri read-only proofs are preserved as reviewable artifacts.
• No customer billing, notification, repository write, or activation occurs during dry-run proof.
• A fresh independent adversarial review finds no unowned state transition, duplicate implementation, or unsupported success path.

Prior Art: What to Adopt and What Not to Claim

Schema reports 98.98% on 25 public ARC-AGI-3 games with a retained Opus/Fable pairing and 95.35% with Sol.
Its released dataset contains 50 retained trajectories, one per game for each pairing, with event logs, session data, snapshots, and executable world models.
The scores are self-reported, use a fixed fallback that keeps the higher per-game run, and have no semi-private or other unseen-set result.

The useful design lesson is concrete: keep an append-only interaction history, write the current hypothesis as an executable artifact, test it against the full history, plan inside it, and stop when reality contradicts it.
For this system, that maps to exact candidate bundles, complete development evidence, executable comparison, and contradiction-driven revision inside the search job.
It does not justify automatic activation or copying a game-specific planning loop into the runtime.

Weco reinforces fixed budgets, candidate lineage, aggressive rejection, heterogeneous tasks, and private checks.
Its reported high rejection rate reinforces the same boundary: autonomous generation may be broad, but activation follows exact measurement and explicit authority.

Tracking

This is the canonical reusable-package issue.

• Intelligence product adoption, tenant behavior, and UI remain tracked in agent-dev-container #2003 and #3214.
• Exact candidate execution source work exists in agent-runtime PR #547.
• The stale full-loop experiment exists in agent-runtime PR #544.
• False Codex failure classification is tracked in traces #36.

The first implementation tranche is order A.
No composition facade starts before that release train is complete.