docs(design): taOS Council (role-benchmarked background model team)#1818
docs(design): taOS Council (role-benchmarked background model team)#1818jaylfc wants to merge 1 commit into
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Warning Review limit reached
Next review available in: 28 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| ``` | ||
|
|
||
| - "Distinct accounts" means distinct signing-key fingerprints, one counted | ||
| submission per fingerprint per (model, role, version). |
There was a problem hiding this comment.
WARNING: N-confirm "distinct accounts" is defeated by unbounded keypair minting
The consensus dedupe keys on the raw signing-key fingerprint (lines 371-372), and submission is gated only by the session cookie (line 347). The hub keystore mints keypairs freely per node (lines 73-74) and the doc never requires a 1:1 binding between an authenticated account and a registered signing key. A single attacker can therefore generate N keypairs and satisfy "N distinct accounts" alone, which directly contradicts the fraud-posture claim that collusion "needs N colluding accounts" (lines 378-380). Require each accepted signing_pubkey to be registered to and owned by the authenticated account (one key per account, or an account-scoped key quota), and verify the cookie's account owns the key on ingest.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| - **On new model.** Notify (source `council`). If the community DB has | ||
| TRUSTED scores for it, offer adoption directly and, when the user has | ||
| enabled auto-adopt in Council settings, adopt without a tap (locked | ||
| decision 4: this is the common case at scale). Otherwise the notification |
There was a problem hiding this comment.
WARNING: Auto-adopt appears to bypass the mandatory consent minting
Lines 241-248 state every Council member is minted through the standard consent path with explicit user approval ("One approval, one canonical id … the registry stays the single source of identity truth"). But auto-adopt "adopts without a tap" (line 454) and "adoption can be automatic with no local gauge" (lines 169-170). If auto-adopt creates a full member plus identity plus registry JWT, it violates that consent invariant; if it only records a trusted score, the term "adoption" is overloaded. Clarify whether auto-adopt mints identity (and if so, that it uses a pre-authorized grant the user enabled, not a silent identity mint) versus merely surfacing trusted scores for later manual approval.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| roles), POST `/api/council/gauges/run`, run view in `CouncilApp.tsx`, tests | ||
| with a stub candidate. | ||
| - Executes model-authored code in worktrees; sandbox posture must be | ||
| reviewed by a maintainer against the app-runtime tiers. |
There was a problem hiding this comment.
WARNING: Harness executes untrusted model output as code without specified isolation
Audition candidates are arbitrary newly surfaced :free models pulled in automatically by the poller (lines 444-456), and S6 executes their outputs as code in worktrees/tmux (line 542). Because candidates can be attacker-submitted, the doc must specify hard isolation for audition runs: no network egress (or strict allowlist), no mounted secrets/keys, ephemeral filesystem, CPU/memory/time limits, and non-privileged execution. Deferring this to "app-runtime tiers" (line 543) is insufficient for untrusted third-party model code; call out the required sandbox guarantees explicitly before S6 is implemented.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| first valid record -> PROVISIONAL (score = that record) | ||
| N distinct accounts, | ||
| all within tolerance -> TRUSTED (score = median of confirming records) | ||
| records disagree |
There was a problem hiding this comment.
SUGGESTION: Median-of-confirming-records may not be integer-storable
gauge_records.score is INTEGER (line 281), but the TRUSTED score is defined as the "median of confirming records" (line 365). With an even number of confirming scores the median can be a .5 value that INTEGER truncation would shift. Define rounding (e.g., round half-up) or store the trusted score separately with fractional precision in gauge_consensus.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| (`tinyagentos/hub/identity.py:165-168`). The artifact bundle (task inputs, | ||
| candidate outputs, grading transcript) is retained locally under | ||
| `data/council/artifacts/{record_id}/` so a score is always auditable; only | ||
| its hash travels. |
There was a problem hiding this comment.
SUGGESTION: Spot-audit of artifacts is not guaranteed
The audit bundle is retained only on the submitter's node (lines 294-296), yet the fraud posture claims taos.my "can request an artifact bundle for a suspicious record" (lines 380-381). If the submitter is offline or deleted the bundle, the audit cannot happen, but the record may remain TRUSTED and even be auto-adopted. Either require server-side retention of artifact bundles for submitted records, or state explicitly that audits are best-effort and document how unauditable-but-trusted records are handled (e.g., re-open to DISPUTED).
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| display_name TEXT NOT NULL, | ||
| description TEXT NOT NULL DEFAULT '', | ||
| gauge_status TEXT NOT NULL, -- 'proven' | 'designed' | 'provisional' | ||
| suite_version TEXT, -- NULL when provisional |
There was a problem hiding this comment.
SUGGESTION: "provisional" is overloaded across two taxonomies
The role registry gauge_status uses 'proven' | 'designed' | 'provisional' (line 217, suite maturity) while the consensus state machine uses UNKNOWN | PROVISIONAL | TRUSTED | DISPUTED (line 363, per-record trust). Same word, different meaning, which will confuse implementers and reviewers. Rename one axis (e.g., registry status ungauged/alpha/graduated, or trust state PENDING).
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
|
|
||
| Tables: `gauge_submissions` (the wire schema plus `signing_pubkey`, verified | ||
| on ingest with the same Ed25519 check as `verify_signature`, | ||
| `tinyagentos/hub/identity.py:212-226`) and `gauge_consensus` |
There was a problem hiding this comment.
SUGGESTION: Clarify which public key is authoritative for verification
W1 says submission is "verified on ingest with the same Ed25519 check as verify_signature" (line 353) and "verification against the hub identity directory record" (line 572), yet the wire schema carries an explicit signing_pubkey (line 340). Specify that the signature is verified against the submitted signing_pubkey AND that this key is the one registered to the authenticated account (closing the first finding's loophole), rather than implying either key alone is sufficient.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 7 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (1 files)
Fix these issues in Kilo Cloud Reviewed by hy3:free · Input: 101.7K · Output: 34.8K · Cached: 243.6K |
1 participant
Summary
Founding design for taOS Council: a background live team of models that monitors what the user does and works together on it. Users add models to the Council; each gets its own registry identity via the existing consent flow. The primary model auditions each member for specific roles, once per model per role, and the signed scores are shared on a public gauge registry at taos.my so only one user globally ever benchmarks a given model. Free-tier models become an instant zero-cost team of specialists (writer, editor, reviewer, summarizer, and more), assigned to projects as standard team members on the board and the a2a channel.
What the doc covers
Design doc only, no code changes.