Skip to content

feat(materials): add OSSF_SCORECARD_JSON material type#3212

Merged
migmartri merged 3 commits into
mainfrom
427-ossf-scorecard-material
Jun 15, 2026
Merged

feat(materials): add OSSF_SCORECARD_JSON material type#3212
migmartri merged 3 commits into
mainfrom
427-ossf-scorecard-material

Conversation

@migmartri

@migmartri migmartri commented Jun 15, 2026

Copy link
Copy Markdown
Member

Description

Adds a dedicated OSSF_SCORECARD_JSON material type so OpenSSF Scorecard results can be attested as first-class, structured evidence instead of opaque SARIF.

The crafter validates the Scorecard V2 JSON report against an embedded JSON Schema (using the existing internal/schemavalidators mechanism, consistent with the OpenAPI/AsyncAPI/CycloneDX materials), uploads the raw report to CAS, and records the Scorecard tool version and aggregate score as material annotations. It supports --no-strict-validation and participates in material auto-detection.

The embedded schema is adapted from OSSF's published json.v2.schema with details/metadata relaxed to allow null, matching real-world Scorecard output.

Closes #427

AI assistance

This change was produced with the assistance of Claude Code (Assisted-by: Claude Code).

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri

Review in cubic

@chainloop-platform

chainloop-platform Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

AI Session Analysis

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 79% 1 ✅ 0 81% AI / 19% Human 17 +699 / -27 1h51m28s

🟡 79% — 81% AI — ✅ All policies passing

Jun 15, 2026 12:25 UTC · 1h51m28s · $45.35 · 127.5k in / 272.5k out · claude-code 2.1.177 (claude-opus-4-8)

View session details ↗

Change Summary

  • Adds OSSF_SCORECARD_JSON material wiring plus JSON Schema validation for Scorecard V2 reports.
  • Adds scorecard crafter fixtures, validator coverage, and a no-strict-validation regression test.
  • Regenerates workflow-contract bindings, JSON Schema artifacts, and CLI docs, then renumbers the enum through rebases.
  • Follows up on bot review comments with a typed-struct parsing change and PR comment replies.

AI Session Overall Score

🟡 79% — Strong implementation, but verification lacks user confirmation and scope drifted into repo guidance.

AI Session Analysis Breakdown

🟢 88% · solution-quality

🟢 Bot feedback was fixed with a typed struct and always-on discriminator, not a bypass. · High Impact

🟢 86% · user-trust-signal

No notes.

🟢 84% · context-and-planning

🟢 User supplied a written spec plus concrete JSON-schema constraints before coding. · High Impact

🟡 76% · verification

🟢 AI added targeted tests and reran them after review fixes and rebases. · High Impact

🟠 Repeated builds and targeted tests passed, but the user never explicitly confirmed the new scorecard flow worked. · Medium Severity

💡 Before closing a user-driven feature, capture one explicit manual confirmation of the shipped behavior.

🟡 70% · alignment

🟠 AI attributed a typed-struct policy to the user, then widened the change into CLAUDE.md guidance without a confirming user turn. · Medium Severity

💡 When a review fix suggests repo-wide guidance, ask explicitly before editing policy docs or claiming the user chose that direction.

🟡 70% · scope-discipline

No notes.


File Attribution

████████████████░░░░ 81% AI / 19% Human

Status Attribution File Lines
modified ai pkg/attestation/crafter/materials/scorecard_test.go +211 / -4
modified ai pkg/attestation/crafter/materials/scorecard.go +167 / -17
created ai internal/schemavalidators/external_schemas/scorecard/scorecard-v2.schema.json +101 / -0
modified ai internal/schemavalidators/schemavalidators.go +45 / -0
created human internal/schemavalidators/testdata/scorecard_valid.json +45 / -0
created human pkg/attestation/crafter/materials/testdata/scorecard-chainloop.json +45 / -0
modified ai internal/schemavalidators/schemavalidators_test.go +36 / -0
created human pkg/attestation/crafter/materials/testdata/scorecard-low.json +25 / -0
created human pkg/attestation/crafter/materials/testdata/scorecard-no-score.json +10 / -0
modified human app/cli/documentation/cli-reference.mdx +3 / -3
modified ai extras/dagger/main.go +2 / -2
modified ai app/controlplane/api/workflowcontract/v1/crafting_schema.proto +3 / -0
modified ai app/cli/cmd/attestation_add.go +1 / -1
modified ai pkg/attestation/crafter/materials/materials.go +2 / -0
modified ai CLAUDE.md +1 / -0
modified ai app/controlplane/api/workflowcontract/v1/crafting_schema_validations.go +1 / -0
created human pkg/attestation/crafter/materials/testdata/scorecard-invalid.json +1 / -0

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-234a5d -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-234a5d -
✅ Passed ai-config-no-secrets ai-coding-session-234a5d -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-234a5d -

Powered by Chainloop and Chainloop Trace

@migmartri migmartri force-pushed the 427-ossf-scorecard-material branch from d339b75 to e0c1b16 Compare June 15, 2026 13:02

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread pkg/attestation/crafter/materials/scorecard.go
@kusari-inspector

kusari-inspector Bot commented Jun 15, 2026

Copy link
Copy Markdown

Kusari Inspector

Kusari Analysis Results:

Proceed with these changes

✅ No Flagged Issues Detected
All values appear to be within acceptable risk parameters.

No pinned version dependency changes, code issues or exposed secrets detected!

Note

View full detailed analysis result for more information on the output and the checks that were run.


@kusari-inspector rerun - Trigger a re-analysis of this PR
@kusari-inspector feedback [your message] - Send feedback to our AI and team
See Kusari's documentation for setup and configuration.
Commit: 9e6f5b9, performed at: 2026-06-15T13:23:50Z

Found this helpful? Give it a 👍 or 👎 reaction!

Comment thread pkg/attestation/crafter/materials/scorecard.go Outdated
@kusari-inspector

Copy link
Copy Markdown

Kusari PR Analysis rerun based on - 9e6f5b9 performed at: 2026-06-15T13:23:51Z - link to updated analysis

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread pkg/attestation/crafter/materials/scorecard.go Outdated
Add a dedicated material type for ingesting OpenSSF Scorecard V2 JSON
reports as first-class evidence. The crafter validates the report against
an embedded JSON Schema, uploads it to CAS, and records the Scorecard
tool version and aggregate score as material annotations.

Closes #427

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: 234a5dc3-baba-4c3d-be42-dbabf15c5487
Address review feedback on the OSSF_SCORECARD_JSON crafter:

- Parse the report into a typed struct instead of accessing fields through
  a generic map[string]interface{}, removing fragile type assertions.
- Always enforce a discriminating-field guard (scorecard.version) so that
  --no-strict-validation no longer lets arbitrary JSON be misclassified as a
  Scorecard report and annotated as such.

Also document the typed-struct-over-empty-interface preference in CLAUDE.md.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: 234a5dc3-baba-4c3d-be42-dbabf15c5487
@migmartri migmartri force-pushed the 427-ossf-scorecard-material branch from 9e6f5b9 to 48344e8 Compare June 15, 2026 14:12
@migmartri migmartri requested a review from a team June 15, 2026 14:15
Address review feedback: the aggregate score annotation was set
unconditionally, so a nonconformant report missing "score" would be
misrepresented as score 0 under --no-strict-validation. Make the score
field a pointer so an absent score is distinguishable from a real 0.0,
and only set the annotation when present.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: 234a5dc3-baba-4c3d-be42-dbabf15c5487
@migmartri migmartri enabled auto-merge (squash) June 15, 2026 14:41
@migmartri migmartri disabled auto-merge June 15, 2026 17:06
@migmartri migmartri merged commit f142a75 into main Jun 15, 2026
16 checks passed
@migmartri migmartri deleted the 427-ossf-scorecard-material branch June 15, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add scorecards support for Chainloop

2 participants