📖 Consolidate v6 docs and add implementation plan#4994
Open
justaugustus wants to merge 9 commits intoossf:mainfrom
Open
📖 Consolidate v6 docs and add implementation plan#4994justaugustus wants to merge 9 commits intoossf:mainfrom
justaugustus wants to merge 9 commits intoossf:mainfrom
Conversation
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
2 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4994 +/- ##
==========================================
+ Coverage 66.80% 69.78% +2.98%
==========================================
Files 230 252 +22
Lines 16602 15736 -866
==========================================
- Hits 11091 10982 -109
+ Misses 4808 3873 -935
- Partials 703 881 +178 🚀 New features to boost your workflow:
|
JamieMagee
reviewed
Apr 4, 2026
JamieMagee
reviewed
Apr 4, 2026
JamieMagee
reviewed
Apr 4, 2026
JamieMagee
reviewed
Apr 4, 2026
JamieMagee
reviewed
Apr 4, 2026
|
This pull request has been marked stale because it has been open for 10 days with no activity |
Signed-off-by: Stephen Augustus <foo@auggie.dev>
Add dependency-ordered implementation plan for Scorecard v6 and fix broken links after docs/v6/ consolidation. Implementation steps: - Step 0: OpenFeature infrastructure (enables all v6 work) - Step 1: Evidence model + framework abstraction (core types) - Step 2: Conformance engine + applicability (core evaluation) - Step 3: Output formats, staggered (JSON → in-toto → Gemara → OSCAL) - Step 4: L1 probe coverage + metadata ingestion (parallel with Steps 2-3) - Step 5: Probe catalog extraction (downstream tool integration) - Steps 6-8: Phase 2 (release integrity, attestation, evidence bundles) - Steps 9-11: Phase 3 (enforcement detection, multi-repo, attestation GA) Key design decisions: - v6 is a clean, backwards-compatible successor (no parallel v5 maintenance) - OpenFeature for granular feature gating during v5→v6 transition - FeatureGate field on checker.Check replaces hard-coded delete list - Feature flag wrapper at internal/featureflags/ (not public API) - Explicit phase gates: Phase 1 must prove value before Phase 2 begins Link fixes: - docs/ROADMAP.md: update proposal and coverage links to docs/v6/ - docs/v6/decisions.md: update coverage link (now same directory) - docs/v6/proposal.md: update coverage link (now same directory) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Restructure Phase 1 to deliver complete OSPS Baseline Level 1 conformance evidence using existing infrastructure where possible. Key changes: **Ordering: Prove abstractions with existing code first** - Step 0: OpenFeature with existing env vars (SCORECARD_V6, SCORECARD_EXPERIMENTAL) - Step 1: Framework abstraction proven with existing checks before building OSPS - Step 2: JSON output extension (use existing format, defer other formats) - Step 3: OSPS Baseline as second framework (uses proven abstraction) - Step 4: Complete L1 coverage (all 9 gap controls closed) **Phase 1 success criteria:** - Complete L1 control coverage (all 9 gap controls + existing coverage validated) - Framework abstraction proven with checks before OSPS Baseline - Production-ready conformance results in extended JSON - Existing checks, probes, scores unchanged (v6 is additive) **Key findings from investigation:** - Probes produce findings (reusable) - Check evaluation logic produces 0-10 scores (NOT reusable for conformance) - Pattern is reusable: "take findings, apply rules, produce verdict" - Don't shoehorn - checks and conformance have different semantics - Metadata ingestion already exists via checks/fileparser/ (no new infrastructure) **Deferred to Phase 2:** - Probe catalog extraction (wait for framework abstraction to stabilize) - Additional output formats (in-toto, Gemara, OSCAL) - Cron infrastructure (storage/serving cost evaluation needed) - Level 2/3 controls, attestation, multi-repo Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Capture recommendations (pending approval) from review discussion: Feature flag changes: - Simplify to two flags: scorecard.experimental and scorecard.v6 - Add feature promotion table for existing flagged features (Webhooks, SBOM, raw format, SARIF must be promoted/migrated in v6) - Remove per-feature granular flags (deferred until actual need arises) - Add testing strategy recommendation (e2e runs twice: default + v6) JSON schema: - Add Option B recommendation: unified evaluations key instead of checks + conformance as parallel top-level keys - Preserve backward compatibility via old schema as default Control catalog: - Replace versioned data file with importing security-baseline Go package - Control definitions from upstream; probe mappings in Scorecard Coverage validation: - Add Step 3.5: human review of L1 coverage analysis before writing probes - Gap probe estimates subject to validated coverage analysis Forge support: - Document GitHub (primary), GitLab (where probes work), Azure DevOps (deferred), local directory (file-based probes only) - Controls unsupported on a forge produce UNKNOWN, not FAIL Baseline levels: - Document that L1/L2/L3 are one framework with levels, not three frameworks Housekeeping: - Remove checkmarks from Phase 1 Complete (not yet done) - Move resolved questions to Resolved decisions section - Remove probe catalog from Phase 1 (already moved to Phase 2) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Add section documenting existing infrastructure that v6 should extend rather than duplicate, based on comprehensive codebase review. - Map execution pipeline integration point (conformance evaluator consumes Result.Findings after probes run, no parallel pipeline needed) - Document 13 reusable components with file locations and how v6 uses each - Identify 4 duplication risks: Framework Result interface may over-abstract, finding.Outcome types may already cover conformance status, applicability could use existing NotApplicable outcome, gap probes may overlap existing Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Integrate Jamie Magee's review feedback on platform capability handling and non-GitHub forge support: Step 1 (Framework abstraction): - Add PlatformCapabilities struct to Framework.Evaluate() interface from day one, avoiding a breaking interface change when non-GitHub forges are introduced - Document ErrUnsupportedFeature inconsistency (license.go falls back, security_policy.go skips, branch_protection.go errors) as a prerequisite for non-GitHub conformance support Step 2 (JSON output): - Add reason field to UNKNOWN statuses in both schema options, preventing schema breakage when non-GitHub forges produce platform-specific UNKNOWNs Phase 2: - Add note that release-related probes (6 L2 controls) will need platform-specific implementations (e.g., Azure Artifacts vs GitHub Releases) - Add ErrUnsupportedFeature standardization as Phase 2 prerequisite New section: - Add Azure DevOps graduation criteria note, to be co-authored with ADO component maintainers when conformance support is scoped Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Integrate Jamie Magee's review feedback on platform observability and release distribution: - Extend UNKNOWN-first honesty principle (design principle #3) to acknowledge that non-GitHub platforms require platform capability metadata to distinguish UNKNOWN from NOT_APPLICABLE, referencing the PlatformCapabilities design in plan.md Step 1 - Add note to Phase 2 that six release-related L2 controls will need platform-specific probe implementations (GitHub Releases vs Azure Artifacts feeds, classic release pipelines, pipeline artifacts) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Three additions to the v6 implementation plan based on codebase research: Step 2 expanded (CLI integration + probe discoverability): - Recommend conformance evaluation runs automatically when scorecard.v6 is enabled, no new --framework flag needed (aligns with proposal architectural target state #5: single run produces both) - Add probe discoverability design: users need to understand probe metadata and remediation content when reading conformance results - Remediation content should reference OSPS Baseline control definitions New Step 5 (probe-eval synchronization hazard fix): - Evaluation functions hardcode expectedProbes lists that must match entries.go; mismatch causes runtime failures - Adding 9+ gap probes in Step 6 without fixing this creates a 3-location synchronization tax per probe - Fix by deriving expectedProbes from probe registry Step renumbering: - Old Step 5 (gap probes) becomes Step 6 - New Step 5 (sync hazard fix) inserted between coverage review and gap probe work Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
Rewrite implementation plan from linear 7-step sequence to 9-step
dependency graph with parallel execution opportunities.
Key structural changes:
- Sync hazard fix moves from Step 5 to Step 1 (benefits all subsequent
steps, not just gap probes)
- Old Step 2 (JSON + CLI + discoverability) split into three independent
steps that can proceed in parallel after framework abstraction:
- Step 3: JSON schema (critical path for OSPS engine)
- Step 4: CLI integration (parallel with Step 3)
- Step 5: Probe discoverability (parallel with Steps 3-6)
- OSPS Baseline engine (Step 6) waits only for framework abstraction
and JSON schema, not for CLI integration or discoverability
Dependency graph:
Step 0 (OpenFeature) → Step 1 (sync fix) → Step 2 (framework)
├─► Step 3 (JSON schema) ──► Step 6 (OSPS engine)
├─► Step 4 (CLI integration) ► Step 7 (review)
└─► Step 5 (discoverability) ► Step 8 (gap probes)
Steps 3, 4, 5 proceed concurrently after Step 2.
Phase 1 completion requires all steps including parallel ones.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Augustus <foo@auggie.dev>
dosubot
bot
added
the
size:XL
justaugustus
commented
Apr 18, 2026
Member
Author
justaugustus
left a comment
justaugustus
left a comment
There was a problem hiding this comment.
@JamieMagee — I've integrated all of your feedback in the most recent commits.
Thanks for doing the first round!
As for your comment about ADO graduation, I'll let you help decide that, now that you're a component owner. :)
Copied your comment to the original ADO feature request, so we don't lose the thread.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Documentation: consolidate Scorecard v6 documents under
docs/v6/and add adependency-ordered implementation plan for Phase 1.
What is the current behavior?
v6 documents are scattered across
openspec/changes/osps-baseline-conformance/and
docs/. There is no implementation plan showing dependency orderingbetween v6 work items.
What is the new behavior (if this is a feature change)?
All v6 documents consolidated under
docs/v6/:proposal.md— architecture and vision (from merged PR 📖 Scorecard v6: OSPS Baseline conformance proposal and 2026 roadmap #4952)decisions.md— reviewer feedback logosps-baseline-coverage.md— control-by-control coverage analysisplan.md— new dependency-ordered implementation planThe implementation plan covers Phase 1 (OSPS Baseline Level 1 conformance
evidence) with 6 steps ordered by dependency:
The plan also includes:
Cross-document links in
ROADMAP.md,decisions.md, andproposal.mdupdated to reflect new file locations.
N/A — documentation only.
Which issue(s) this PR fixes
NONE
Special notes for your reviewer
This is a follow-up to PR #4952 (merged). The proposal and decisions documents
are unchanged except for link fixes. The implementation plan (
plan.md) is newcontent. Several recommendations are marked "pending approval" for Steering
Committee discussion.
Does this PR introduce a user-facing change?