Skip to content

chore: enforce tidy/manifests checks in test CI#1888

Merged
moshloop merged 3 commits intomainfrom
chore/ci-manifests-tidy-checks
Apr 15, 2026
Merged

chore: enforce tidy/manifests checks in test CI#1888
moshloop merged 3 commits intomainfrom
chore/ci-manifests-tidy-checks

Conversation

@adityathebe
Copy link
Copy Markdown
Member

@adityathebe adityathebe commented Apr 15, 2026

  • add pre-test go mod tidy check and fail on go.mod/go.sum drift
  • run make manifests before tests in the test workflow
  • improve handwritten schema error context and include Playwright deepcopy generation update

Summary by CodeRabbit

  • New Features

    • Added comprehensive schema generation for configuration change types with embedded examples and automatic documentation.
    • Introduced automated migration testing and benchmarking capabilities.
  • Improvements

    • Refactored configuration change schema structure for enhanced clarity and validation.
  • Chores

    • Enhanced CI/CD verification workflows.
    • Updated build and testing infrastructure.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 15, 2026

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

The changes implement automated schema generation for ConfigChangeDetail variants by introducing a new Go generator program, refactoring the schema output process to use generated JSON with embedded examples, adding GitHub Actions verification workflow, and exporting internal types for public use in schema generation.

Changes

Cohort / File(s) Summary
GitHub Actions CI/CD
.github/workflows/verify-generated.yaml
Added new workflow to verify generated artifacts on push to main and pull requests. Runs go mod tidy, make manifests, and enforces repository cleanliness via git diff.
Build System & Automation
Makefile, Taskfile.yaml, .gitignore
Updated Makefile with new test-migrate and test-bench targets; added Taskfile with corresponding automation tasks for running migration E2E tests and benchmarks with base comparisons; added ignore patterns for generated binaries.
Schema Generator - Core Implementation
hack/generate-schemas/change_types.go, change_types_test.go
Added new schema generator program that builds JSON Schema for ConfigChangeDetail variants with discriminator-based union via oneOf, injects required kind properties, and embeds examples from types; includes comprehensive test verifying schema structure, variants, and examples.
Schema Generator - Support Code
hack/generate-schemas/main.go
Refactored main generator to replace handwritten schema workflow with dynamic generation; added dedicated call to new generateChangeTypesSchema; switched error handling from fatalf to returning wrapped errors via RunE.
Schema Generation Infrastructure
schema/openapi/generate.go, schema/openapi/openapi.go
Introduced GenerateSchemaWithCommentsFrom exported function enabling schema generation with doc comments sourced from external packages; updated //go:embed directive to embed only *.json files.
Generated Output
schema/openapi/change-types.schema.json
Refactored from handwritten to fully generated schema; inlined all ConfigChangeDetail variant definitions under $defs with discriminator-based oneOf union, removed type aliases, and relaxed field constraints; reorganized metadata positioning.
Type System Exports & Examples
types/config_changes.go, types/config_changes_examples.go, types/config_changes_test.go
Exported ConfigChangeDetailTypes slice and added new ConfigChangeExamples slice with realistic sample instances for all variants; updated tests to use exported slice.
Generated Code
shell/zz_generated.deepcopy.go
Added deep-copy handling for ExecSetup.Playwright field to allocate and copy referenced RuntimeSetup values.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer
    participant Make as Build System
    participant Gen as Schema Generator
    participant Types as Type System
    participant Schema as Schema Output
    participant CI as GitHub Actions
    
    Dev->>Make: Execute `make manifests`
    Make->>Gen: Run hack/generate-schemas
    Gen->>Types: Read ConfigChangeDetailTypes<br/>& ConfigChangeExamples
    Types-->>Gen: Return all variants with examples
    Gen->>Gen: Reflect types to JSON Schema<br/>Build discriminator oneOf<br/>Inject kind properties
    Gen->>Schema: Write change-types.schema.json
    Schema-->>Gen: Success
    
    Dev->>Dev: Commit changes
    Dev->>CI: Push to main/open PR
    CI->>Make: Run `go mod tidy`
    Make-->>CI: Verify go.mod/go.sum clean
    CI->>Make: Run `make manifests`
    Make->>Gen: Generate schemas
    Gen->>Schema: Output schema
    CI->>CI: Check git diff --exit-code
    alt Changes detected
        CI-->>Dev: ❌ Workflow fails:<br/>Re-run make manifests locally
    else No changes
        CI-->>Dev: ✅ Verification passed
    end
Loading
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 65.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the main objective: enforcing tidy and manifests checks in the CI test workflow. It directly reflects the primary intent stated in the PR description.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch chore/ci-manifests-tidy-checks
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch chore/ci-manifests-tidy-checks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 15, 2026

Benchstat (RLS)

Base: 10fddf4c480c14a468bdc836ae147c98c1909d60
Head: ba94348a87e9c0da0d50b1fbca76d2b93f43e250

📊 8 minor regression(s) (all within 5% threshold)

Benchmark Base Head Change p-value
RLS/Sample-15000/config_changes/Without_RLS-4 5.365m 5.567m +3.76% 0.004
RLS/Sample-15000/configs/Without_RLS-4 7.089m 7.351m +3.69% 0.002
RLS/Sample-15000/config_summary/Without_RLS-4 61.30m 63.46m +3.51% 0.041
RLS/Sample-15000/config_detail/Without_RLS-4 3.913m 4.045m +3.39% 0.002
RLS/Sample-15000/config_classes/Without_RLS-4 3.314m 3.402m +2.67% 0.002
RLS/Sample-15000/config_names/Without_RLS-4 12.87m 13.14m +2.14% 0.002
RLS/Sample-15000/config_detail/With_RLS-4 123.4m 125.5m +1.76% 0.002
RLS/Sample-15000/config_classes/With_RLS-4 123.6m 125.5m +1.58% 0.002
✅ 3 improvement(s)
Benchmark Base Head Change p-value
RLS/Sample-15000/catalog_changes/Without_RLS-4 5.582m 5.383m -3.56% 0.041
RLS/Sample-15000/change_types/With_RLS-4 5.667m 5.487m -3.19% 0.026
RLS/Sample-15000/config_changes/With_RLS-4 127.9m 127.2m -0.59% 0.041
Full benchstat output
goos: linux
goarch: amd64
pkg: github.com/flanksource/duty/bench
cpu: AMD EPYC 7763 64-Core Processor                
                                               │ bench-base.txt │          bench-head.txt           │
                                               │     sec/op     │   sec/op     vs base              │
RLS/Sample-15000/catalog_changes/Without_RLS-4      5.582m ± 5%   5.383m ± 1%  -3.56% (p=0.041 n=6)
RLS/Sample-15000/catalog_changes/With_RLS-4         128.6m ± 4%   129.2m ± 1%       ~ (p=0.065 n=6)
RLS/Sample-15000/config_changes/Without_RLS-4       5.365m ± 2%   5.567m ± 4%  +3.76% (p=0.004 n=6)
RLS/Sample-15000/config_changes/With_RLS-4          127.9m ± 1%   127.2m ± 1%  -0.59% (p=0.041 n=6)
RLS/Sample-15000/config_detail/Without_RLS-4        3.913m ± 1%   4.045m ± 2%  +3.39% (p=0.002 n=6)
RLS/Sample-15000/config_detail/With_RLS-4           123.4m ± 1%   125.5m ± 2%  +1.76% (p=0.002 n=6)
RLS/Sample-15000/config_names/Without_RLS-4         12.87m ± 2%   13.14m ± 1%  +2.14% (p=0.002 n=6)
RLS/Sample-15000/config_names/With_RLS-4            126.2m ± 4%   125.9m ± 0%       ~ (p=0.394 n=6)
RLS/Sample-15000/config_summary/Without_RLS-4       61.30m ± 4%   63.46m ± 1%  +3.51% (p=0.041 n=6)
RLS/Sample-15000/config_summary/With_RLS-4          740.9m ± 1%   743.0m ± 0%       ~ (p=0.180 n=6)
RLS/Sample-15000/configs/Without_RLS-4              7.089m ± 2%   7.351m ± 1%  +3.69% (p=0.002 n=6)
RLS/Sample-15000/configs/With_RLS-4                 124.9m ± 1%   123.8m ± 0%       ~ (p=0.065 n=6)
RLS/Sample-15000/analysis_types/Without_RLS-4       3.944m ± 1%   3.927m ± 2%       ~ (p=0.240 n=6)
RLS/Sample-15000/analysis_types/With_RLS-4          3.948m ± 1%   3.948m ± 2%       ~ (p=0.937 n=6)
RLS/Sample-15000/analyzer_types/Without_RLS-4       3.728m ± 2%   3.733m ± 1%       ~ (p=0.818 n=6)
RLS/Sample-15000/analyzer_types/With_RLS-4          3.783m ± 2%   3.800m ± 2%       ~ (p=0.485 n=6)
RLS/Sample-15000/change_types/Without_RLS-4         5.283m ± 6%   5.393m ± 3%       ~ (p=0.310 n=6)
RLS/Sample-15000/change_types/With_RLS-4            5.667m ± 3%   5.487m ± 3%  -3.19% (p=0.026 n=6)
RLS/Sample-15000/config_classes/Without_RLS-4       3.314m ± 1%   3.402m ± 1%  +2.67% (p=0.002 n=6)
RLS/Sample-15000/config_classes/With_RLS-4          123.6m ± 0%   125.5m ± 0%  +1.58% (p=0.002 n=6)
RLS/Sample-15000/config_types/Without_RLS-4         3.994m ± 1%   3.990m ± 1%       ~ (p=0.937 n=6)
RLS/Sample-15000/config_types/With_RLS-4            125.3m ± 0%   125.6m ± 1%       ~ (p=0.093 n=6)
geomean                                             19.38m        19.53m       +0.76%

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 15, 2026

Benchstat (Other)

Base: 10fddf4c480c14a468bdc836ae147c98c1909d60
Head: ba94348a87e9c0da0d50b1fbca76d2b93f43e250

✅ 1 improvement(s)
Benchmark Base Head Change p-value
ResourceSelectorQueryBuild/tags-4 17.13µ 16.97µ -0.88% 0.015
Full benchstat output
goos: linux
goarch: amd64
pkg: github.com/flanksource/duty/bench
cpu: AMD EPYC 7763 64-Core Processor                
                                                       │ bench-base.txt │          bench-head.txt           │
                                                       │     sec/op     │   sec/op     vs base              │
InsertionForRowsWithAliases/external_users.aliases-4       599.4µ ±  5%   589.7µ ± 4%       ~ (p=0.699 n=6)
InsertionForRowsWithAliases/config_items.external_id-4     1.109m ± 14%   1.107m ± 9%       ~ (p=0.699 n=6)
ResourceSelectorConfigs/name-4                             212.7µ ±  3%   211.1µ ± 1%       ~ (p=0.180 n=6)
ResourceSelectorConfigs/name_and_type-4                    231.3µ ±  5%   230.4µ ± 5%       ~ (p=0.589 n=6)
ResourceSelectorConfigs/tags-4                             27.57m ±  3%   28.05m ± 3%       ~ (p=0.180 n=6)
ResourceSelectorQueryBuild/name-4                          43.18µ ±  1%   43.24µ ± 0%       ~ (p=1.000 n=6)
ResourceSelectorQueryBuild/name_and_type-4                 63.25µ ±  1%   63.21µ ± 1%       ~ (p=1.000 n=6)
ResourceSelectorQueryBuild/tags-4                          17.13µ ±  1%   16.97µ ± 1%  -0.88% (p=0.015 n=6)
geomean                                                    283.9µ         283.2µ       -0.25%

@adityathebe
Copy link
Copy Markdown
Member Author

@moshloop

Error: unable to save handwritten schema (src: change-types.handwritten.schema.json, dst: ../../schema/openapi/change-types.schema.json): unable to read handwritten schema change-types.handwritten.schema.json: open change-types.handwritten.schema.json: no such file or directory

@adityathebe adityathebe requested a review from moshloop April 15, 2026 11:05
@moshloop moshloop force-pushed the chore/ci-manifests-tidy-checks branch from 264e5f9 to 1fabac2 Compare April 15, 2026 17:45
@moshloop moshloop enabled auto-merge (rebase) April 15, 2026 18:21
adityathebe and others added 3 commits April 15, 2026 21:21
…f handwritten

Replace handwritten change-types schema with automated generation via reflection and post-processing. This ensures the schema stays in sync with ConfigChangeDetail variants and their examples.

Changes:
- Add hack/generate-schemas/change_types.go with schema generation logic that reflects the changeDetailsEnvelope, injects kind discriminators, and produces a stable JSON output
- Add hack/generate-schemas/change_types_test.go with comprehensive tests for schema generation
- Add types/config_changes_examples.go with representative examples for all ConfigChangeDetail variants
- Export ConfigChangeDetailTypes and ConfigChangeExamples for use by the schema generator
- Update main.go to call generateChangeTypesSchema instead of copying handwritten schema
- Regenerate schema/openapi/change-types.schema.json with new examples and stable key ordering
- Add Taskfile.yaml with test:migrate and test:bench tasks for local development
- Update .gitignore to exclude generated binaries

The generated schema now includes per-variant examples and maintains deterministic output across runs.
…ustom comment sources

Refactor schema generation to support loading Go doc comments from a different package than the reflected object. This enables code generators and similar tools in main packages to produce schemas with comments from library packages.

Introduces GenerateSchemaWithCommentsFrom() as the new public API while keeping GenerateSchema() as a convenience wrapper. The internal generateSchema() function now handles both cases via a commentsFrom parameter.

Also fixes embed directive in openapi.go to be more specific (*.json instead of *).
@moshloop moshloop force-pushed the chore/ci-manifests-tidy-checks branch from 01935f1 to ba94348 Compare April 15, 2026 18:21
@moshloop moshloop merged commit 03a5b64 into main Apr 15, 2026
17 of 18 checks passed
@moshloop moshloop deleted the chore/ci-manifests-tidy-checks branch April 15, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants