Skip to content

Coverage and gap report comparing Policy requirements against evaluation artifacts #27

Description

@jpower432

Is your feature request related to a problem? Please describe.

After generating Rego policies for in-scope controls, consumers have no way to see which controls have enforcement and which don't. They need a coverage report answering: "Of the 14 controls in my Policy, how many have Rego policies, how many pass, and which are gaps?" Today this requires manually comparing the Policy's requirements against files in the policy/ directory.

Describe the solution you'd like

A coverage/gap report tool that compares a Policy's in-scope controls against existing enforcement artifacts and test results, producing a structured summary with three buckets:

  • Implemented, passing -- enforcement artifact exists, tests pass
  • Implemented, failing -- enforcement artifact exists, tests fail (remediation needed)
  • Not implemented (gap) -- no enforcement artifact exists (build or accept risk)

Inputs:

  • Policy artifact (or effective policy name from MCP server)
  • Policy directory path (where .rego files live)
  • Optionally: existing EvaluationLog to determine pass/fail status

Describe alternatives you've considered

  • Agent-guided gap analysis via SKILL.md: The skill instructs the agent to list Policy requirements, scan the policy/ directory, and report gaps. Non-deterministic and depends on agent file system traversal accuracy.
  • CI-only coverage report: A complypack coverage CLI command that runs in pipelines. Useful but doesn't integrate with the MCP-based authoring workflow.
  • Manual tracking: Does not scale and falls out of sync immediately.\

Additional context

Open question: how does the system determine a control is "implemented"? Options include presence of a .rego file matching the control ID pattern, an existing EvaluationLog entry, or explicit declaration in the Policy. This needs design resolution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions