feat(cli): auto-collect AI agent config during attestation init#2843
Merged
Conversation
Add a Collector interface to the crafter package that enables pluggable auto-discovery of evidence during attestation init. Refactor existing PR metadata collection into PRMetadataCollector and introduce a new AIAgentConfigCollector that discovers Claude agent config files (CLAUDE.md, .claude/*, .mcp.json) and attaches them as EVIDENCE material. Ref: PFM-4919 Signed-off-by: Jose Ignacio Paris <jiparis@chainloop.dev> Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
jiparis
commented
Mar 13, 2026
| } | ||
| tmpFile.Close() | ||
|
|
||
| if _, err := cr.AddMaterialContractFree(ctx, attestationID, "EVIDENCE", "ai-agent-config-claude", tmpFile.Name(), casBackend, nil); err != nil { |
Member
Author
There was a problem hiding this comment.
I'm open to suggestions. This is storing the aggregated agent config. I was thinking about also uploading individual files to the CAS so that they can be referenced by their digest.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
8 issues found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="app/cli/pkg/action/attestation_init.go">
<violation number="1" location="app/cli/pkg/action/attestation_init.go:304">
P2: Registering collectors inside `Run` accumulates them across repeated invocations, so later runs execute the same collectors multiple times.</violation>
</file>
<file name="pkg/attestation/crafter/collector_prmetadata.go">
<violation number="1" location="pkg/attestation/crafter/collector_prmetadata.go:72">
P3: Use a `*` in the temp-file pattern so the recorded artifact filename keeps its `.json` suffix.</violation>
</file>
<file name="internal/aiagentconfig/builder.go">
<violation number="1" location="internal/aiagentconfig/builder.go:40">
P0: Reject symlinks before reading discovered config files. A matching symlink currently lets a repo upload arbitrary host files during auto-collection.</violation>
<violation number="2" location="internal/aiagentconfig/builder.go:52">
P2: Include the relative path in `ConfigHash`; hashing only file contents misses renames and moves of path-sensitive config files.</violation>
</file>
<file name="internal/aiagentconfig/discover_test.go">
<violation number="1" location="internal/aiagentconfig/discover_test.go:47">
P2: Normalize expected paths instead of hard-coding `/`; these assertions will fail on Windows because `Discover` returns OS-native separators.</violation>
</file>
<file name="pkg/attestation/crafter/crafter.go">
<violation number="1" location="pkg/attestation/crafter/crafter.go:145">
P1: Return immediately when reloading crafting state fails; continuing can run collectors against an empty state and panic.</violation>
</file>
<file name="internal/aiagentconfig/discover.go">
<violation number="1" location="internal/aiagentconfig/discover.go:44">
P2: Joining `rootDir` into the glob pattern makes worktrees whose path contains glob metacharacters (`*`, `?`, `[`) discover the wrong files or fail entirely.</violation>
<violation number="2" location="internal/aiagentconfig/discover.go:45">
P2: `filepath.Glob` suppresses directory read errors here, so unreadable AI config paths will be silently omitted from the collected evidence.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
- Reject symlinks before reading config files to prevent arbitrary file upload - Return early in RunCollectors when crafting state reload fails - Include file path in config hash to detect renames/moves - Fix temp file pattern to preserve .json suffix Signed-off-by: Jose Ignacio Paris <jiparis@chainloop.dev> Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
1 issue found across 4 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/aiagentconfig/builder.go">
<violation number="1" location="internal/aiagentconfig/builder.go:41">
P1: This only blocks leaf symlinks; files under symlinked `.claude` directories can still escape `rootDir` and be uploaded.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
…directories The previous symlink check only caught leaf file symlinks. A symlinked parent directory (e.g., .claude/ -> /outside/dir) could still allow reading arbitrary files. Now uses filepath.EvalSymlinks to resolve the full path and verifies it remains under rootDir. Signed-off-by: Jose Ignacio Paris <jiparis@chainloop.dev> Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/aiagentconfig/builder.go">
<violation number="1" location="internal/aiagentconfig/builder.go:105">
P2: This prefix check rejects every valid file when the resolved root directory is `/`. Use a relative-path containment check instead of appending a separator and comparing raw prefixes.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
… JSON schema validation Replace the generic EVIDENCE material type with a dedicated CHAINLOOP_AI_AGENT_CONFIG type for AI agent configuration data. This adds JSON schema validation when users manually add materials via `chainloop attestation add`, matching the CHAINLOOP_PR_INFO pattern. Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
5 issues found across 21 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="app/controlplane/api/workflowcontract/v1/crafting_schema_validations.go">
<violation number="1" location="app/controlplane/api/workflowcontract/v1/crafting_schema_validations.go:52">
P2: Add tests for the new `CHAINLOOP_AI_AGENT_CONFIG` auto-detection path in both crafter and policy-eval flows.
(Based on your team's feedback about adding or updating tests for new paths.) [FEEDBACK_USED]</violation>
</file>
<file name="internal/schemavalidators/internal_schemas/aiagentconfig/ai-agent-config-1.0.schema.json">
<violation number="1" location="internal/schemavalidators/internal_schemas/aiagentconfig/ai-agent-config-1.0.schema.json:7">
P1: This schema validates the inner payload, but the emitted evidence document wraps that payload under `data`, so the actual JSON will not match this schema.</violation>
<violation number="2" location="internal/schemavalidators/internal_schemas/aiagentconfig/ai-agent-config-1.0.schema.json:92">
P2: These new fields are effectively unvalidated because they only have descriptions and no schema constraints.</violation>
</file>
<file name="app/controlplane/api/workflowcontract/v1/crafting_schema.proto">
<violation number="1" location="app/controlplane/api/workflowcontract/v1/crafting_schema.proto:153">
P2: This new enum value diverges from the emitted attestation shape. The collector records AI agent config as `EVIDENCE` with `chainloop.material.evidence.id=CHAINLOOP_AI_AGENT_CONFIG`, so exposing `CHAINLOOP_AI_AGENT_CONFIG` as a separate `MaterialType` makes contracts and policies able to reference a type the CLI never produces.</violation>
</file>
<file name="pkg/attestation/crafter/materials/chainloop_ai_agent_config.go">
<violation number="1" location="pkg/attestation/crafter/materials/chainloop_ai_agent_config.go:57">
P2: Validate the original JSON payload instead of a re-marshaled struct; this flow strips unsupported fields before schema validation, so malformed AI-agent evidence can be accepted.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| CraftingSchema_Material_ZAP_DAST_ZIP, | ||
| CraftingSchema_Material_SLSA_PROVENANCE, | ||
| CraftingSchema_Material_CHAINLOOP_RUNNER_CONTEXT, | ||
| CraftingSchema_Material_CHAINLOOP_AI_AGENT_CONFIG, |
There was a problem hiding this comment.
P2: Add tests for the new CHAINLOOP_AI_AGENT_CONFIG auto-detection path in both crafter and policy-eval flows.
(Based on your team's feedback about adding or updating tests for new paths.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At app/controlplane/api/workflowcontract/v1/crafting_schema_validations.go, line 52:
<comment>Add tests for the new `CHAINLOOP_AI_AGENT_CONFIG` auto-detection path in both crafter and policy-eval flows.
(Based on your team's feedback about adding or updating tests for new paths.) </comment>
<file context>
@@ -49,6 +49,7 @@ var CraftingMaterialInValidationOrder = []CraftingSchema_Material_MaterialType{
CraftingSchema_Material_ZAP_DAST_ZIP,
CraftingSchema_Material_SLSA_PROVENANCE,
CraftingSchema_Material_CHAINLOOP_RUNNER_CONTEXT,
+ CraftingSchema_Material_CHAINLOOP_AI_AGENT_CONFIG,
CraftingSchema_Material_ATTESTATION,
CraftingSchema_Material_CONTAINER_IMAGE,
</file context>
Change schema version from 1.0 to 0.1 to reflect the experimental nature of the feature using semantic versioning without patch. Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
…ctors PR metadata collection remains always-on. Other collectors like AI agent config require explicit opt-in via --collectors flag (e.g. --collectors aiconfig). Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
1 issue found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/aiagentconfig/builder_test.go">
<violation number="1" location="internal/aiagentconfig/builder_test.go:53">
P2: This test expects the wrong AI agent config schema version; `Build` emits `0.1`, not `v1alpha`.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
2 issues found across 8 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/schemavalidators/testdata/ai_agent_config_minimal.json">
<violation number="1" location="internal/schemavalidators/testdata/ai_agent_config_minimal.json:11">
P2: This fixture's `sha256` and `size` do not match its `base64_content`, so the new "valid minimal config" sample is not a real AI-agent-config payload.</violation>
</file>
<file name="internal/schemavalidators/testdata/ai_agent_config_valid.json">
<violation number="1" location="internal/schemavalidators/testdata/ai_agent_config_valid.json:17">
P2: This "valid" fixture's `sha256` and `size` do not match the embedded `base64_content`, so it no longer represents a payload the builder can actually emit.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
migmartri
approved these changes
Mar 13, 2026
|
|
||
| // Collector auto-discovers and attaches evidence during attestation init. | ||
| // Each collector runs best-effort: failures are logged but never fail the attestation. | ||
| type Collector interface { |
migmartri
reviewed
Mar 13, 2026
migmartri
left a comment
migmartri
left a comment
Member
There was a problem hiding this comment.
Review done with the help of Claude. Comments below focus on medium-severity items.
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
4 issues found across 12 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/schemavalidators/testdata/ai_agent_config_valid.json">
<violation number="1" location="internal/schemavalidators/testdata/ai_agent_config_valid.json:19">
P2: This fixture now validates `content`, but the documented AI agent evidence format uses `base64_content`. That drifts the schema tests away from the attestation contract and can break compatibility with real collector output.
(Based on your team's feedback about shared config/API compatibility.) [FEEDBACK_USED]</violation>
</file>
<file name="app/cli/cmd/attestation_init.go">
<violation number="1" location="app/cli/cmd/attestation_init.go:80">
P2: This collector validation is bypassed whenever version auto-discovery returns early, so mistyped `--collectors` values still get silently skipped.</violation>
</file>
<file name="internal/schemavalidators/internal_schemas/aiagentconfig/ai-agent-config-0.1.schema.json">
<violation number="1" location="internal/schemavalidators/internal_schemas/aiagentconfig/ai-agent-config-0.1.schema.json:69">
P1: Keeping schema version `0.1` while renaming required `base64_content` to `content` makes the evidence format backward-incompatible and will reject payloads that still follow the documented envelope.
(Based on your team's feedback about ensuring cross-component and version compatibility.) [FEEDBACK_USED]</violation>
</file>
<file name="internal/aiagentconfig/aiagentconfig.go">
<violation number="1" location="internal/aiagentconfig/aiagentconfig.go:36">
P1: This renames the evidence JSON field from `base64_content` to `content`, which breaks the documented `0.1` schema and existing consumers.
(Based on your team's feedback about ensuring cross-component and version compatibility.) [FEEDBACK_USED]</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
3 issues found across 5 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/aiagentconfig/discover_test.go">
<violation number="1" location="internal/aiagentconfig/discover_test.go:73">
P2: This test only covers a single nested directory, so it won't catch that the current glob-based implementation still misses deeper `.cursor/rules/**` files.</violation>
<violation number="2" location="internal/aiagentconfig/discover_test.go:172">
P3: Use a strict equality assertion here so the explicit empty-map cases don't also pass on a nil result.
(Based on your team's feedback about preferring explicit states over opaque nils.) [FEEDBACK_USED]</violation>
</file>
<file name="internal/aiagentconfig/discover.go">
<violation number="1" location="internal/aiagentconfig/discover.go:43">
P2: These `**` patterns will not recurse with `filepath.Glob`, so deeper nested Cursor rules are silently missed.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/aiagentconfig/discover.go">
<violation number="1" location="internal/aiagentconfig/discover.go:43">
P1: This only scans one nested level under `.cursor/rules`, so deeper valid Cursor rule files are silently skipped.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
new CHAINLOOP_AI_CONFIG material: Add
AIAgentConfigCollectorthat discovers Claude agent configuration files (CLAUDE.md, .claude/*, .mcp.json) in the working directoryinitial support for Claude and Cursor file patterns
The evidence JSON follows the custom evidence envelope format with
0.1schema version, containing file contents (base64), SHA-256 digests, git context, and a combined config hash for drift detectionIntroduce a
Collectorinterface in the crafter package for pluggable auto-discovery of evidence during attestation initRefactor existing PR/MR metadata collection into a
PRMetadataCollectorimplementing the new interfaceAdd
--collectorcommand line option to enable additional collectors.aiconfigwill enable AIAgentConfigCollectorAttestation:
> cldev att init --workflow aiworkflow --project my-project --replace --collectors aiconfig ... INF discovered AI agent config files files=7 DBG crafting file backend=AWS-S3 digest=sha256:39cd2a1a04d3b5636d07c9b858f2687e5ddde1d93d096170b9f7068c8bee633e filename=ai-agent-config-985068682.json max_size=300M path=/var/folders/ls/cv3k03v57ns18mmwjjbgy8z00000gn/T/ai-agent-config-985068682.json size=46.9K skip_upload=false DBG uploading backend=AWS-S3 INF uploading ai-agent-config-985068682.json - sha256:39cd2a1a04d3b5636d07c9b858f2687e5ddde1d93d096170b9f7068c8bee633e ... ┌───────────────────────────┬──────────────────────────────────────┐ │ Initialized At │ 13 Mar 26 11:30 UTC │ ├───────────────────────────┼──────────────────────────────────────┤ │ Attestation ID │ 9863b0e2-d650-4386-a9c0-69df9aacf90b │ │ Organization │ my-org │ │ Name │ aiworkflow │ │ Project │ my-project │ │ Version │ v1.81.2+next (prerelease) │ │ Contract │ my-project-aiworkflow (revision 1) │ │ Policy violation strategy │ ENFORCED │ └───────────────────────────┴──────────────────────────────────────┘ ┌────────────────────────────────────────────────────────────────────────────────────┐ │ Materials │ ├──────────┬─────────────────────────────────────────────────────────────────────────┤ │ Name │ ai-agent-config-claude │ │ Type │ CHAINLOOP_AI_AGENT_CONFIG │ │ Set │ Yes │ │ Required │ No │ │ Value │ ai-agent-config-985068682.json │ │ Digest │ sha256:39cd2a1a04d3b5636d07c9b858f2687e5ddde1d93d096170b9f7068c8bee633e │ └──────────┴─────────────────────────────────────────────────────────────────────────┘ ➜ cldev att push ... INF push completed ┌───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Initialized At │ 13 Mar 26 11:30 UTC │ ├───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Attestation ID │ 9863b0e2-d650-4386-a9c0-69df9aacf90b │ │ Digest │ sha256:e253b4aab0780c13dee41a0551defb1ccf7c433d9c6bf15834b6a33850477d27 │ │ Organization │ my-org │ │ Name │ aiworkflow │ │ Project │ my-project │ │ Version │ v1.81.2+next (prerelease) │ │ Contract │ my-project-aiworkflow (revision 1) │ │ Policy violation strategy │ ENFORCED │ │ Attestation View URL │ http://localhost:3000/u/my-org/workflow-runs/sha256:e253b4aab0780c13dee41a0551defb1ccf7c433d9c6bf15834b6a33850477d27 │ └───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────────────────────────────────────┐ │ Materials │ ├──────────┬─────────────────────────────────────────────────────────────────────────┤ │ Name │ ai-agent-config-claude │ │ Type │ CHAINLOOP_AI_AGENT_CONFIG │ │ Set │ Yes │ │ Required │ No │ │ Value │ ai-agent-config-985068682.json │ │ Digest │ sha256:39cd2a1a04d3b5636d07c9b858f2687e5ddde1d93d096170b9f7068c8bee633e │ └──────────┴─────────────────────────────────────────────────────────────────────────┘Evidence example:
{ "schema_version": "0.1", "agent": { "name": "claude" }, "config_hash": "b4ae27822a36662a969abebe34b758f742bf69eca506977d6cd867dc0f4043bd", "captured_at": "2026-03-13T08:45:51Z", "config_files": [ { "path": ".claude/settings.json", "sha256": "95125ab3c3c69aa1754e55e62dddf654ca08878902ecae024aa0c15c44840eeb", "size": 258, "base64_content": "..." }, { "path": ".claude/skills/custom-builtin-functions/SKILL.md", "sha256": "1c5a493d38d5e2f74c16098f38a71fb5f1c90b923bd2e33cace435e14893b2cc", "size": 2289, "base64_content": "..." }, { "path": ".claude/skills/dependabot-pr-automation/SKILL.md", "sha256": "0c10891c284f495b62d2dd795bf461290298a9f9b4a8210fa48e9b45dd982142", "size": 3743, "base64_content": "..." }, { "path": ".claude/skills/upgrading-chart/SKILL.md", "sha256": "4e984c9d3cfcefcf3ec3ec0fa2665de7d58bd2c15c66240579f4e14c6a90ed23", "size": 4409, "base64_content": "..." }, { "path": ".claude/skills/upgrading-golang/SKILL.md", "sha256": "939f590c2cf0c35b59b9afd52981e3e5c61aa4c0555d3472e7cd371ec069d8db", "size": 3888, "base64_content": "..." }, { "path": ".claude/skills/vulnerability-remediation/SKILL.md", "sha256": "1180d8f0d868d175ad27791b7dc9ff0d7fe045c59b52fe95a217cbc88c754268", "size": 7541, "base64_content": "..." }, { "path": "CLAUDE.md", "sha256": "e1714f0232211ace49916a368c244df73948626485e39d1197a2b90091cf3040", "size": 12832, "base64_content": "..." } ] }Ref: PFM-4919