Summary
• Elevate our existing test suite into an official RDCP Conformance Kit that:
◦ Validates any RDCP server implementation against the protocol
◦ Dogfoods and gates RDCP-SDK changes
◦ Serves as living, executable documentation in the wiki
• Package it as an npm CLI, Docker image, and GitHub Action, with discovery-driven, capability-aware selection and machine-readable compliance reports.
Motivation
• Our tests already encode the RDCP spec and guided TDD for the SDK. Formalizing them as a spec-as-code kit:
◦ Provides an objective, repeatable compliance signal to users and vendors
◦ Prevents drift between spec, docs, and implementation
◦ Accelerates adoption (bootstrap templates + CI integration)
◦ Improves internal speed and safety (pre-release gates)
Objectives
• Black-box conformance tests: exercise only HTTP endpoints, no repo-internal imports
• Capability-aware selection: auto-read /.well-known/rdcp and only run relevant tests
• Profiles: basic | standard | enterprise and optional capabilities (multi-tenancy, TTL, audit, rate-limit, JWKS)
• Parametrized auth fixtures (api-key, bearer scopes/issuer/audience, mTLS certs)
• Deterministic timing for TTL/rate-limit; stable audit assertions
• Rich reports: JUnit, JSON, Markdown with pass/fail per requirement ID
• Multiple runtimes: npm CLI, Docker, GitHub Action
• Traceability: map each test to protocol requirement IDs; generate compliance matrix
Non-goals
• Validating non-RDCP application-specific routes
• Locking implementations to one tech stack
• Replacing formal spec text (this complements the spec)
Scope and Profiles
• Security levels: basic (api-key), standard (bearer), enterprise (mTLS + token)
• Capabilities toggles: multiTenancy, temporaryControls/TTL, rateLimit, auditTrail, tokenLifecycle+JWKS, metrics
• Conformance runner reads /.well-known/rdcp → selects suites based on declared level + capabilities
Approach
- Black-box tests
• Ensure conformance suites use only HTTP requests; no monorepo internals
- Discovery-driven runner
• Fetch /.well-known/rdcp; derive active profiles/capabilities; skip non-applicable tests
- Test tagging
• Tag tests by profile/capability (e.g., [basic], [standard][jwt], [enterprise][mtls], [tenant], [ttl], [audit], [rate-limit], [jwks])
- Parametrized auth fixtures
• Provide a single config input for keys/tokens/certs and tenant IDs
- Deterministic timing
• Use controlled windows or buffers for rate-limit and TTL; avoid flake
- Stable assertions for audit
• Prefer structured responses/headers for audit checks; if logs are used, define minimal stable JSON line format
- Reporting
• Emit:
◦ JUnit (CI consumption)
◦ JSON (machine aggregation)
◦ Markdown (human-readable compliance report with requirement IDs)
- Packaging
• npm: rdcp-conformance
• Docker: mojoatomic/rdcp-conformance:latest
• GitHub Action: rdcp/conformance@v1 (composite/action)
- Versioning and traceability
• Protocol requirement IDs (e.g., REQ-DISC-001) mapped to test titles
• Runner prints protocol version and test-kit version; publish compatibility table
Roadmap (Phased)
M1 — Extraction and tagging
Split tests: tests/conformance/** (server), tests/sdk/** (client-only)
Replace repo imports with pure HTTP; add tags per capability/profile
Add discovery-gating utility and test selection layer
Introduce config schema (YAML/JSON)
M2 — Runner + reports
Build CLI: rdcp-conformance (config, discovery URL, profile override)
JUnit + JSON + Markdown reporting with requirement mapping
Deterministic timing utilities for TTL/rate-limit
Stable audit assertion helpers
M3 — Packaging and CI integration
npm publish (runner)
Docker image (minimal Node base)
GitHub Action (inputs: baseUrl, credentials, profile)
Badges: “RDCP Conformant: Basic/Standard/Enterprise”, sub-badges by capability
M4 — Docs and samples
Wiki: How to run locally (curl + CLI), in CI (Action), with Docker
Templates for user projects (config examples for each security level)
Compliance report examples and interpretation guide
Traceability matrix (spec ↔ tests)
Deliverables
• Conformance CLI + Docker + GitHub Action
• Test tagging and discovery-driven selection
• Reporting artifacts (JUnit/JSON/Markdown)
• Wiki documentation and templates
• Compliance badges and version compatibility table
Acceptance criteria
• Running against our demo app produces 100% pass for declared capabilities; skips are clearly indicated
• Running against a partial implementation passes only implemented capabilities with 0 false positives
• Reports include requirement coverage, pass/fail, and links to spec sections
• Action usable in external repos with one-page setup
Risks and mitigations
• Timing flake (TTL/rate-limit): use buffers and clear windows; allow retry-once for borderline windows
• Audit coupling to logs: prefer structured response or headers; if logs, standardize minimal JSON schema
• Profile/capability drift: treat /.well-known/rdcp as source of truth; include “strict mode” flag to fail on misdeclared capabilities
• Spec evolution: versioned requirement IDs; deprecation policy with dual-run mode
Developer experience (examples)
• Local CLI
bash
• Docker
bash
• GitHub Action
yaml
Dogfooding plan (this repo)
• Add a pre-release CI job that runs the conformance kit against the demo app and adapters (Express/Fastify/Koa)
• Gate merges to main by conformance (for declared profiles/capabilities)
• Publish compliance report artifact on each release; embed badge in README/wiki
Adoption in user projects
• Provide a “getting started” config template for each security level/profile
• Encourage teams to run the Action in their CI; start with “basic” and expand to “standard/enterprise”
• Offer a sample tenant matrix for multi-tenancy validation
• Provide a minimal “self-check” target that vendors can publish in their pipelines
Open questions
• Should we allow “soft fail” (warning) for optional features or strictly treat them as skipped?
• Do we require a minimum set of capabilities per profile for the badge (e.g., standard must implement bearer + scopes)?
• Preferred audit verification channel: header/response vs. log line; do we standardize an Audit endpoint?
Opinion
• This implementation will bootstrap and dogfood future development both here and in user projects. It codifies the spec into an executable, portable standard, accelerates vendor adoption, and provides strong safety rails for the SDK’s evolution.
Summary
• Elevate our existing test suite into an official RDCP Conformance Kit that:
◦ Validates any RDCP server implementation against the protocol
◦ Dogfoods and gates RDCP-SDK changes
◦ Serves as living, executable documentation in the wiki
• Package it as an npm CLI, Docker image, and GitHub Action, with discovery-driven, capability-aware selection and machine-readable compliance reports.
Motivation
• Our tests already encode the RDCP spec and guided TDD for the SDK. Formalizing them as a spec-as-code kit:
◦ Provides an objective, repeatable compliance signal to users and vendors
◦ Prevents drift between spec, docs, and implementation
◦ Accelerates adoption (bootstrap templates + CI integration)
◦ Improves internal speed and safety (pre-release gates)
Objectives
• Black-box conformance tests: exercise only HTTP endpoints, no repo-internal imports
• Capability-aware selection: auto-read /.well-known/rdcp and only run relevant tests
• Profiles: basic | standard | enterprise and optional capabilities (multi-tenancy, TTL, audit, rate-limit, JWKS)
• Parametrized auth fixtures (api-key, bearer scopes/issuer/audience, mTLS certs)
• Deterministic timing for TTL/rate-limit; stable audit assertions
• Rich reports: JUnit, JSON, Markdown with pass/fail per requirement ID
• Multiple runtimes: npm CLI, Docker, GitHub Action
• Traceability: map each test to protocol requirement IDs; generate compliance matrix
Non-goals
• Validating non-RDCP application-specific routes
• Locking implementations to one tech stack
• Replacing formal spec text (this complements the spec)
Scope and Profiles
• Security levels: basic (api-key), standard (bearer), enterprise (mTLS + token)
• Capabilities toggles: multiTenancy, temporaryControls/TTL, rateLimit, auditTrail, tokenLifecycle+JWKS, metrics
• Conformance runner reads /.well-known/rdcp → selects suites based on declared level + capabilities
Approach
• Ensure conformance suites use only HTTP requests; no monorepo internals
• Fetch /.well-known/rdcp; derive active profiles/capabilities; skip non-applicable tests
• Tag tests by profile/capability (e.g., [basic], [standard][jwt], [enterprise][mtls], [tenant], [ttl], [audit], [rate-limit], [jwks])
• Provide a single config input for keys/tokens/certs and tenant IDs
• Use controlled windows or buffers for rate-limit and TTL; avoid flake
• Prefer structured responses/headers for audit checks; if logs are used, define minimal stable JSON line format
• Emit:
◦ JUnit (CI consumption)
◦ JSON (machine aggregation)
◦ Markdown (human-readable compliance report with requirement IDs)
• npm: rdcp-conformance
• Docker: mojoatomic/rdcp-conformance:latest
• GitHub Action: rdcp/conformance@v1 (composite/action)
• Protocol requirement IDs (e.g., REQ-DISC-001) mapped to test titles
• Runner prints protocol version and test-kit version; publish compatibility table
Roadmap (Phased)
M1 — Extraction and tagging
Split tests: tests/conformance/** (server), tests/sdk/** (client-only)
Replace repo imports with pure HTTP; add tags per capability/profile
Add discovery-gating utility and test selection layer
Introduce config schema (YAML/JSON)
M2 — Runner + reports
Build CLI: rdcp-conformance (config, discovery URL, profile override)
JUnit + JSON + Markdown reporting with requirement mapping
Deterministic timing utilities for TTL/rate-limit
Stable audit assertion helpers
M3 — Packaging and CI integration
npm publish (runner)
Docker image (minimal Node base)
GitHub Action (inputs: baseUrl, credentials, profile)
Badges: “RDCP Conformant: Basic/Standard/Enterprise”, sub-badges by capability
M4 — Docs and samples
Wiki: How to run locally (curl + CLI), in CI (Action), with Docker
Templates for user projects (config examples for each security level)
Compliance report examples and interpretation guide
Traceability matrix (spec ↔ tests)
Deliverables
• Conformance CLI + Docker + GitHub Action
• Test tagging and discovery-driven selection
• Reporting artifacts (JUnit/JSON/Markdown)
• Wiki documentation and templates
• Compliance badges and version compatibility table
Acceptance criteria
• Running against our demo app produces 100% pass for declared capabilities; skips are clearly indicated
• Running against a partial implementation passes only implemented capabilities with 0 false positives
• Reports include requirement coverage, pass/fail, and links to spec sections
• Action usable in external repos with one-page setup
Risks and mitigations
• Timing flake (TTL/rate-limit): use buffers and clear windows; allow retry-once for borderline windows
• Audit coupling to logs: prefer structured response or headers; if logs, standardize minimal JSON schema
• Profile/capability drift: treat /.well-known/rdcp as source of truth; include “strict mode” flag to fail on misdeclared capabilities
• Spec evolution: versioned requirement IDs; deprecation policy with dual-run mode
Developer experience (examples)
• Local CLI
bash
• Docker
bash
• GitHub Action
yaml
Dogfooding plan (this repo)
• Add a pre-release CI job that runs the conformance kit against the demo app and adapters (Express/Fastify/Koa)
• Gate merges to main by conformance (for declared profiles/capabilities)
• Publish compliance report artifact on each release; embed badge in README/wiki
Adoption in user projects
• Provide a “getting started” config template for each security level/profile
• Encourage teams to run the Action in their CI; start with “basic” and expand to “standard/enterprise”
• Offer a sample tenant matrix for multi-tenancy validation
• Provide a minimal “self-check” target that vendors can publish in their pipelines
Open questions
• Should we allow “soft fail” (warning) for optional features or strictly treat them as skipped?
• Do we require a minimum set of capabilities per profile for the badge (e.g., standard must implement bearer + scopes)?
• Preferred audit verification channel: header/response vs. log line; do we standardize an Audit endpoint?
Opinion
• This implementation will bootstrap and dogfood future development both here and in user projects. It codifies the spec into an executable, portable standard, accelerates vendor adoption, and provides strong safety rails for the SDK’s evolution.