From 51237b1aa059d065c06d236141729accdfcd91a2 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 14 May 2026 10:27:17 -0300 Subject: [PATCH 01/37] Add design plan for spot check jobs --- docs/plans/spot-check-jobs.md | 439 ++++++++++++++++++++++++++++++++++ 1 file changed, 439 insertions(+) create mode 100644 docs/plans/spot-check-jobs.md diff --git a/docs/plans/spot-check-jobs.md b/docs/plans/spot-check-jobs.md new file mode 100644 index 000000000..2c467f8f0 --- /dev/null +++ b/docs/plans/spot-check-jobs.md @@ -0,0 +1,439 @@ +# Spot-Check Jobs in Component Readiness + +## Context + +Component Readiness is built around statistical analysis of **test pass rates** across +many runs of multiple prow job configurations. The sample is typically the last 7 days +and the basis is the 30 days prior to the last release's GA — a very stable period. +Fisher's Exact Test (or pass-rate thresholds) determines whether a regression is +statistically significant. + +We need to also monitor **rarely-run spot-check jobs** that verify obscure +configurations (vSphere hybrid installs, CPU partitioning, etcd scaling, etc.). These +jobs are run once and retried up to three times if they fail. The requirement is +simple: **the job must pass at least once in a 30-day window**. There is no basis +comparison, no Fisher's Exact, no confidence/minFail/pity thresholds — just a binary +"did it pass." + +These results must appear in the existing Component Readiness report grid, be +drillable via test details, and participate in regression tracking and triage. + +### Existing "Rarely Run" Concept + +There is an existing `VariantJunitTableOverride` mechanism +(`pkg/apis/config/v1/types.go:30-44`) that was intended for rarely-run jobs. It pulls +junit results from alternate BigQuery tables with extended date ranges. This is not +what we want — it is still based on test pass rates within rarely-run jobs, which is +too strict. We want **full job pass**, at least once, across a few retries. + +The existing `JobTier: rare` value will be replaced with `JobTier: spotcheck`. + +## Design + +### 1. New Job Variants + +Three changes to the variant system identify spot-check jobs: + +**New variant constants** in `pkg/variantregistry/ocp.go`: + +```go +const ( + VariantSpotCheckComponent = "SpotCheckComponent" + VariantSpotCheckCapability = "SpotCheckCapability" +) +``` + +**New setter function** `setSpotCheckComponent` registered in `IdentifyVariants` +**before** `setJobTier`. When matched, it atomically sets all three variants: + +```go +func setSpotCheckComponent(_ logrus.FieldLogger, variants map[string]string, jobName string) { + spotCheckPatterns := []struct { + substrings []string + component string + capability string + }{ + {[]string{"-cpu-partitioning"}, "Node", "CPU Partitioning"}, + {[]string{"-etcd-scaling"}, "etcd", "Scaling"}, + {[]string{"-vsphere-hybrid"}, "Installer", "vSphere Hybrid"}, + } + + for _, entry := range spotCheckPatterns { + if allSubstringsMatch(jobName, entry.substrings) { + variants[VariantSpotCheckComponent] = entry.component + variants[VariantSpotCheckCapability] = entry.capability + variants[VariantJobTier] = "spotcheck" + return + } + } +} +``` + +**Remove `"rare"` patterns** from `setJobTier` (lines 743-744). Jobs that were `rare` +either become `spotcheck` (via the new setter) or stay `candidate` until promoted. + +**Add to `importantVariants`** in `pkg/testidentification/ocp_variants.go` so they +flow through to the `job_variants` BigQuery table. + +**JobTier lifecycle:** + +``` +candidate → spotcheck + ↑ + add pattern to setSpotCheckComponent in ocp.go +``` + +### 2. View Configuration: Spot-Check Sample Window + +Views define a separate 30-day spot-check sample window. The normal 7-day test sample +window is unsuitable because spot-check jobs run infrequently. + +**New field in `pkg/apis/api/componentreport/crview/types.go`:** + +```go +type View struct { + // ... existing fields ... + SpotCheckSample *reqopts.RelativeRelease `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` +} +``` + +**Example in `config/views.yaml`:** + +```yaml +- name: 5.0-main + sample_release: + release: "5.0" + relative_start: now-7d + relative_end: now + spot_check_sample: + release: "5.0" + relative_start: now-30d + relative_end: now +``` + +**Carry resolved times through to `pkg/apis/api/componentreport/reqopts/types.go`:** + +```go +type RequestOptions struct { + // ... existing fields ... + SpotCheckSample *Release `json:"spot_check_sample,omitempty"` +} +``` + +**Resolve in `pkg/api/componentreadiness/utils/queryparamparser.go`:** +In `ParseComponentReportRequest`, after resolving view defaults (~line 50): + +```go +if view != nil && view.SpotCheckSample != nil { + resolved, err := GetViewReleaseOptions(releases, "spot_check", *view.SpotCheckSample, crTimeRoundingFactor) + if err != nil { + return + } + opts.SpotCheckSample = &resolved +} +``` + +Views **do not** add `"spotcheck"` to their `JobTier` include list. Normal test +queries filter on `JobTier: [blocking, informing, standard]`, which naturally excludes +spot-check jobs. The spot-check middleware independently queries with its own window. + +### 3. New Middleware: `spotcheckjobs` + +**Location:** `pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go` + +Implements `middleware.Middleware` (interface at +`pkg/api/componentreadiness/middleware/interface.go:15-40`). + +```go +type SpotCheckJobs struct { + dataProvider dataprovider.DataProvider + reqOptions reqopts.RequestOptions + log logrus.FieldLogger + + // Populated during QueryTestDetails, consumed by PreTestDetailsAnalysis + sampleJobDetails map[string][]dataprovider.JobRunDetail +} +``` + +#### Query Phase — Inject Synthetic Test Results + +Queries the BQ `jobs` table (joined with `job_variants`) for all jobs with +`JobTier = 'spotcheck'` in the spot-check sample window. Applies normal view variant +filters (Platform, Architecture, etc.) except JobTier. Groups results by +(SpotCheckComponent, SpotCheckCapability, column variant combo). + +Each group becomes one synthetic test result injected via `sampleStatusCh`: + +- **At least one pass:** `TestStatus{TotalCount: 1, SuccessCount: 1}` → green +- **No passes:** `TestStatus{TotalCount: 1, SuccessCount: 0}` → red + +No basis data is injected. The `Component` and `Capabilities` fields on `TestStatus` +come from the `SpotCheckComponent` / `SpotCheckCapability` variant values, placing the +synthetic test in the correct row of the CR grid. + +The synthetic test ID is deterministic: +`spotcheck::` + +The synthetic test name: +`[spot-check] / must pass at least once per sample window` + +#### PreAnalysis Phase — Bypass Fisher's, Set Status Directly + +```go +func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, + testStats *testdetails.TestComparison) error { + + if !strings.HasPrefix(testKey.TestID, "spotcheck:") { + return nil + } + + if testStats.SampleStats.SuccessCount > 0 { + testStats.ReportStatus = crtest.NotSignificant + testStats.Explanations = append(testStats.Explanations, + "Spot-check job passed at least once in the sample window") + } else { + testStats.ReportStatus = crtest.ExtremeRegression + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Spot-check job did not pass in the %d-day sample window", ...)) + } + + testStats.Comparison = crtest.SpotCheck + testStats.AnalysisComplete = true // signals assessComponentStatus to skip + return nil +} +``` + +This completely bypasses Fisher's Exact, confidence thresholds, minFail, pity factor, +and basis comparison. The middleware directly determines the outcome. + +#### QueryTestDetails Phase — Load Job Run Details + +Queries individual job runs (not aggregated) for spot-check jobs, storing them for +`PreTestDetailsAnalysis` to inject as synthetic `TestJobRunRows`. + +#### PreTestDetailsAnalysis Phase — Inject Job Runs for Drill-Down + +Injects the cached job run rows into `status.SampleStatus` so the test details page +renders individual job runs with pass/fail and prow links. + +#### PostAnalysis Phase — No-op + +The `RegressionTracker` middleware handles regression injection. The +`RegressionAllowances` middleware should skip spot-check tests (they have no basis). + +### 4. AnalysisComplete Flag + +**New field in `pkg/apis/api/componentreport/testdetails/types.go`:** + +```go +type TestComparison struct { + // ... existing fields ... + AnalysisComplete bool `json:"-"` +} +``` + +**Guard in `pkg/api/componentreadiness/component_report.go`:** + +```go +func (c *ComponentReportGenerator) assessComponentStatus( + testStats *testdetails.TestComparison, logger *log.Entry) { + if testStats.AnalysisComplete { + return + } + // ... existing Fisher's / pass-rate logic unchanged ... +} +``` + +This is a minimal change that allows any middleware to fully control analysis outcome. + +### 5. New Comparison Type + +**New constant in `pkg/apis/api/componentreport/crtest/types.go`:** + +```go +const ( + PassRate Comparison = "pass_rate" + FisherExact Comparison = "fisher_exact" + SpotCheck Comparison = "spot_check" // new +) +``` + +### 6. New BigQuery Queries + +**New methods on `DataProvider` interface +(`pkg/api/componentreadiness/dataprovider/interface.go`):** + +```go +type SpotCheckQuerier interface { + QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + start, end time.Time) ([]SpotCheckGroup, error) + + QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + spotCheckComponent, spotCheckCapability string, + variants map[string]string, + start, end time.Time) ([]JobRunDetail, error) +} + +type SpotCheckGroup struct { + SpotCheckComponent string + SpotCheckCapability string + Variants map[string]string + TotalRuns int + SuccessfulRuns int + JobNames []string +} + +type JobRunDetail struct { + JobName string + RunID string + URL string + StartTime time.Time + Success bool +} +``` + +**Component report query** (`QuerySpotCheckJobRuns`): + +```sql +SELECT + jv_SpotCheckComponent.variant_value AS spot_check_component, + jv_SpotCheckCapability.variant_value AS spot_check_capability, + -- dynamic column group by variants (Platform, Arch, Network, etc.) + COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, + COUNT(DISTINCT IF(jobs.prowjob_state = 'success', + jobs.prowjob_build_id, NULL)) AS successful_runs, + ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names +FROM {dataset}.jobs jobs + LEFT JOIN {dataset}.job_variants jv_SpotCheckComponent + ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name + AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent' + LEFT JOIN {dataset}.job_variants jv_SpotCheckCapability + ON jobs.prowjob_job_name = jv_SpotCheckCapability.job_name + AND jv_SpotCheckCapability.variant_name = 'SpotCheckCapability' + LEFT JOIN {dataset}.job_variants jv_Release ... + -- other variant joins for column group by +WHERE jobs.prowjob_start >= @From + AND jobs.prowjob_start < @To + AND jv_Release.variant_value = @Release + AND jv_SpotCheckComponent.variant_value IS NOT NULL + -- view include_variant filters (Platform, Architecture, etc.) +GROUP BY spot_check_component, spot_check_capability, variant_Platform, ... +``` + +Hits only the `jobs` table (joined with `job_variants`), never `junit`. Fast and cheap. + +**Test details query** (`QuerySpotCheckJobRunDetails`): Returns individual job runs +with pass/fail for a specific component/capability/variant combo. + +### 7. Middleware Registration + +In `pkg/api/componentreadiness/component_report.go`, `initializeMiddleware()`: + +```go +func (c *ComponentReportGenerator) initializeMiddleware() { + c.middlewares = middleware.List{} + + if c.ReqOptions.SpotCheckSample != nil { + c.middlewares = append(c.middlewares, + spotcheckjobs.NewSpotCheckJobsMiddleware(c.dataProvider, c.ReqOptions)) + } + + // ... existing middleware (releasefallback, regressiontracker, regressionallowances, linkinjector) ... +} +``` + +Spot-check middleware runs first so its synthetic results are in place before other +middleware processes them. + +### 8. Regression Tracking + +Spot-check regressions participate in the existing regression tracking pipeline +(`pkg/api/componentreadiness/regressiontracker.go:271-404`, +`SyncRegressionsForReport`). This works automatically because: + +- Synthetic spot-check tests appear in `report.Rows[].Columns[].RegressedTests` + like any other regressed test. +- `SyncRegressionsForReport` iterates all regressed tests and calls + `backend.OpenRegression(view, regTest)` for new ones. +- The synthetic test ID (`spotcheck:installer:vsphere-hybrid`) is stable and + deterministic, so regression records persist correctly across report runs. +- The `RegressionTracker` middleware (`PostAnalysis`) applies triage status + (triaged, fixed, failed-fixed) to spot-check regressions the same as normal ones. + +No changes needed to the regression tracking code. + +### 9. Frontend: `spot_check` Comparison Type + +**`sippy-ng/src/component_readiness/CompReadyTestDetailRow.js`:** + +The test details row currently renders pass_rate/successes/failures/flakes in the +`infoCell` function (lines 54-67). For `comparison === "spot_check"`, render a simpler +view: + +- Total job runs attempted +- Successful runs +- Required passes (1) +- No Fisher's Exact column +- No basis stats column + +**`sippy-ng/src/component_readiness/TestDetailsReport.js`:** + +When `data.analyses[0].comparison === "spot_check"`: +- Hide Fisher's Exact confidence display +- Hide basis release stats section +- Show explanation text prominently ("Spot-check job did not pass in 30-day window") +- Job run table still renders normally with prow links + +This is a small conditional — the page structure stays the same. + +## Files Changed + +| File | Change | +|------|--------| +| `pkg/variantregistry/ocp.go` | Add `setSpotCheckComponent` setter, new constants, remove `"rare"` patterns from `setJobTier`, register setter before `setJobTier` | +| `pkg/testidentification/ocp_variants.go` | Add `SpotCheckComponent`, `SpotCheckCapability` to `importantVariants` | +| `pkg/apis/api/componentreport/crview/types.go` | Add `SpotCheckSample *RelativeRelease` to `View` | +| `pkg/apis/api/componentreport/reqopts/types.go` | Add `SpotCheckSample *Release` to `RequestOptions` | +| `pkg/apis/api/componentreport/testdetails/types.go` | Add `AnalysisComplete bool` to `TestComparison` | +| `pkg/apis/api/componentreport/crtest/types.go` | Add `SpotCheck` comparison constant | +| `pkg/api/componentreadiness/dataprovider/interface.go` | Add `SpotCheckQuerier` interface, `SpotCheckGroup`, `JobRunDetail` types | +| `pkg/api/componentreadiness/dataprovider/bigquery/provider.go` | Implement `QuerySpotCheckJobRuns`, `QuerySpotCheckJobRunDetails` | +| `pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go` | **New file** — middleware implementation | +| `pkg/api/componentreadiness/component_report.go` | Add `AnalysisComplete` guard in `assessComponentStatus`, register middleware in `initializeMiddleware` | +| `pkg/api/componentreadiness/utils/queryparamparser.go` | Resolve `SpotCheckSample` from view config | +| `config/views.yaml` | Add `spot_check_sample` to relevant views | +| `sippy-ng/src/component_readiness/CompReadyTestDetailRow.js` | Conditional rendering for `spot_check` comparison | +| `sippy-ng/src/component_readiness/TestDetailsReport.js` | Hide Fisher/basis sections for `spot_check` | + +## What Doesn't Change + +- Fisher's Exact / pass-rate analysis logic (guarded by `AnalysisComplete`) +- Report types (`ComponentReport`, `ReportRow`, `ReportColumn`, `ReportTestSummary`) +- URL param structure (`testId`, `component`, `capability`, variant params all work) +- Caching strategy +- Regression tracking code (spot-check tests participate automatically) + +## Verification + +1. **Variant mapping:** After adding patterns to `setSpotCheckComponent`, run the + variant loader and verify jobs get `SpotCheckComponent`, `SpotCheckCapability`, + and `JobTier: spotcheck` variants in BigQuery's `job_variants` table. + +2. **Component report:** Load a view with `spot_check_sample` configured. Verify + spot-check tests appear in the correct component/capability rows. Break a + spot-check job (or test with a job that has no passes in the window) and verify + it shows as an extreme regression. + +3. **Test details drill-down:** Click a spot-check regression in the CR grid. Verify + the test details page shows individual job runs with prow links, no Fisher's + Exact stats, and the spot-check explanation text. + +4. **Regression tracking:** Verify a failing spot-check job creates a + `test_regressions` record. Verify triaging the regression via the normal triage + flow works (status changes to triaged/fixed/failed-fixed). + +5. **No interference:** Verify normal test-based regressions are unaffected — + Fisher's Exact still runs, basis comparison still works, existing views show the + same results. From 2a17450848e6c5a974be84b4d7cbcd4cfcd27108 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 14 May 2026 14:41:49 -0300 Subject: [PATCH 02/37] Broken first claude attempt at spot check jobs --- config/views.yaml | 4 + .../componentreadiness/component_report.go | 10 + .../dataprovider/bigquery/provider.go | 220 ++++++++++++++ .../dataprovider/interface.go | 35 +++ .../dataprovider/postgres/provider.go | 13 + .../middleware/spotcheckjobs/spotcheckjobs.go | 283 ++++++++++++++++++ .../utils/queryparamparser.go | 9 + pkg/apis/api/componentreport/crtest/types.go | 1 + pkg/apis/api/componentreport/crview/types.go | 5 + pkg/apis/api/componentreport/reqopts/types.go | 4 + .../api/componentreport/testdetails/types.go | 4 + pkg/bigquery/bqlabel/labels.go | 2 + .../CompReadyTestDetailRow.js | 64 ++-- .../component_readiness/CompReadyTestPanel.js | 41 ++- .../component_readiness/TestDetailsReport.js | 7 +- 15 files changed, 658 insertions(+), 44 deletions(-) create mode 100644 pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go diff --git a/config/views.yaml b/config/views.yaml index cfab679f3..b15414b95 100755 --- a/config/views.yaml +++ b/config/views.yaml @@ -9,6 +9,10 @@ component_readiness: release: "5.0" relative_start: now-7d relative_end: now + spot_check_sample: + release: "5.0" + relative_start: now-30d + relative_end: now variant_options: column_group_by: Network: { } diff --git a/pkg/api/componentreadiness/component_report.go b/pkg/api/componentreadiness/component_report.go index 6c09bacd6..e1c30505e 100644 --- a/pkg/api/componentreadiness/component_report.go +++ b/pkg/api/componentreadiness/component_report.go @@ -17,6 +17,7 @@ import ( "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/linkinjector" regressionallowances2 "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/regressionallowances" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/regressiontracker" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/spotcheckjobs" "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" @@ -276,6 +277,12 @@ func (c *ComponentReportGenerator) getCache() cache.Cache { func (c *ComponentReportGenerator) initializeMiddleware() { c.middlewares = middleware.List{} + + // Spot-check jobs middleware runs first so synthetic results are in place for other middleware. + if c.ReqOptions.SpotCheckSample != nil { + c.middlewares = append(c.middlewares, spotcheckjobs.NewSpotCheckJobsMiddleware(c.dataProvider, c.ReqOptions)) + } + // Initialize all our middleware applicable to this request. if c.ReqOptions.AdvancedOption.IncludeMultiReleaseAnalysis && c.ReqOptions.SampleRelease.PullRequestOptions == nil { c.middlewares = append(c.middlewares, releasefallback.NewReleaseFallbackMiddleware(c.dataProvider, c.ReqOptions, c.releaseConfigs)) @@ -760,6 +767,9 @@ func getRegressionStatus(basisPassPercentage, samplePassPercentage float64) crte // (fishers, pass rate, bayes (future)) and the middlewares (fallback, intentional regressions, // cross variant compare, rarely run jobs, etc.) func (c *ComponentReportGenerator) assessComponentStatus(testStats *testdetails.TestComparison, logger *log.Entry) { + if testStats.AnalysisComplete { + return + } // Catch unset required confidence, typically unit tests opts := c.ReqOptions.AdvancedOption if testStats.RequiredConfidence == 0 { diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 85d97523f..4fb1d532a 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -374,6 +374,226 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string return variants, nil } +// --- SpotCheckQuerier --- + +func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { + + columnGroupByVariants := reqOptions.VariantOption.ColumnGroupBy + if len(columnGroupByVariants) == 0 { + columnGroupByVariants = sets.NewString("Platform", "Architecture", "Network") + } + + var selectVariants, joinVariants string + var groupByParts []string + for _, v := range columnGroupByVariants.List() { + cleanV := param.Cleanse(v) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanV, cleanV, cleanV, v) + selectVariants += fmt.Sprintf("jv_%s.variant_value AS variant_%s,\n", cleanV, cleanV) + groupByParts = append(groupByParts, fmt.Sprintf("jv_%s.variant_value", cleanV)) + } + groupByVariants := strings.Join(groupByParts, ", ") + + // Always join Release and JobTier + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_Release ON jobs.prowjob_job_name = jv_Release.job_name AND jv_Release.variant_name = 'Release'\n", + p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", + p.client.Dataset) + + // Track which variant groups already have JOINs + joinedVariants := sets.NewString(columnGroupByVariants.List()...) + joinedVariants.Insert("Release", "JobTier") + + // Build include variant filters (Platform, Architecture, etc.) but skip JobTier + variantFilters := "" + var params []bigquery.QueryParameter + includeVariants := reqOptions.VariantOption.IncludeVariants + if includeVariants == nil { + includeVariants = map[string][]string{} + } + for _, group := range sortedKeys(includeVariants) { + if group == "JobTier" { + continue + } + cleanGroup := param.Cleanse(group) + if !joinedVariants.Has(group) { + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + joinedVariants.Insert(group) + } + paramName := fmt.Sprintf("variantGroup_%s", cleanGroup) + variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: includeVariants[group], + }) + } + + queryString := fmt.Sprintf(` + SELECT + %s + COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, + COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, + ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names + FROM %s.jobs jobs + %s + WHERE jobs.prowjob_start >= DATETIME(@From) + AND jobs.prowjob_start < DATETIME(@To) + AND jv_Release.variant_value = @Release + AND jv_JobTier.variant_value = 'rare' + AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') + %s + GROUP BY %s + `, selectVariants, p.client.Dataset, joinVariants, variantFilters, + groupByVariants) + + params = append(params, + bigquery.QueryParameter{Name: "From", Value: start}, + bigquery.QueryParameter{Name: "To", Value: end}, + bigquery.QueryParameter{Name: "Release", Value: reqOptions.SampleRelease.Name}, + ) + + q := p.client.Query(ctx, bqlabel.CRSpotCheck, queryString) + q.Parameters = params + + it, err := q.Read(ctx) + if err != nil { + return nil, fmt.Errorf("error executing spot check query: %w", err) + } + + var results []dataprovider.SpotCheckGroup + for { + var rawRow map[string]bigquery.Value + err := it.Next(&rawRow) + if err == iterator.Done { + break + } + if err != nil { + return nil, fmt.Errorf("error reading spot check row: %w", err) + } + + group := dataprovider.SpotCheckGroup{ + Variants: map[string]string{}, + } + for _, v := range columnGroupByVariants.List() { + cleanV := param.Cleanse(v) + if val, ok := rawRow["variant_"+cleanV]; ok && val != nil { + group.Variants[v] = val.(string) + } + } + if val, ok := rawRow["total_runs"]; ok && val != nil { + group.TotalRuns = int(val.(int64)) + } + if val, ok := rawRow["successful_runs"]; ok && val != nil { + group.SuccessfulRuns = int(val.(int64)) + } + if val, ok := rawRow["job_names"]; ok && val != nil { + for _, jn := range val.([]bigquery.Value) { + group.JobNames = append(group.JobNames, jn.(string)) + } + } + results = append(results, group) + } + + log.Infof("spot check query returned %d groups", len(results)) + return results, nil +} + +func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + variants map[string]string, + start, end time.Time) ([]dataprovider.JobRunDetail, error) { + + joinVariants := fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_Release ON jobs.prowjob_job_name = jv_Release.job_name AND jv_Release.variant_name = 'Release'\n", + p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", + p.client.Dataset) + + variantFilters := "" + var params []bigquery.QueryParameter + for k, v := range variants { + cleanK := param.Cleanse(k) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanK, cleanK, cleanK, k) + paramName := fmt.Sprintf("variant_%s", cleanK) + variantFilters += fmt.Sprintf(" AND jv_%s.variant_value = @%s", cleanK, paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: v, + }) + } + + queryString := fmt.Sprintf(` + SELECT + jobs.prowjob_job_name AS job_name, + jobs.prowjob_build_id AS run_id, + jobs.prowjob_url AS url, + jobs.prowjob_start AS start_time, + (jobs.prowjob_state = 'success') AS success + FROM %s.jobs jobs + %s + WHERE jobs.prowjob_start >= DATETIME(@From) + AND jobs.prowjob_start < DATETIME(@To) + AND jv_Release.variant_value = @Release + AND jv_JobTier.variant_value = 'rare' + AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') + %s + ORDER BY jobs.prowjob_start DESC + `, p.client.Dataset, joinVariants, variantFilters) + + params = append(params, + bigquery.QueryParameter{Name: "From", Value: start}, + bigquery.QueryParameter{Name: "To", Value: end}, + bigquery.QueryParameter{Name: "Release", Value: reqOptions.SampleRelease.Name}, + ) + + q := p.client.Query(ctx, bqlabel.CRSpotCheckDetails, queryString) + q.Parameters = params + + it, err := q.Read(ctx) + if err != nil { + return nil, fmt.Errorf("error executing spot check details query: %w", err) + } + + type detailRow struct { + JobName string `bigquery:"job_name"` + RunID string `bigquery:"run_id"` + URL string `bigquery:"url"` + StartTime time.Time `bigquery:"start_time"` + Success bool `bigquery:"success"` + } + + var results []dataprovider.JobRunDetail + for { + var row detailRow + err := it.Next(&row) + if err == iterator.Done { + break + } + if err != nil { + return nil, fmt.Errorf("error reading spot check detail row: %w", err) + } + results = append(results, dataprovider.JobRunDetail{ + JobName: row.JobName, + RunID: row.RunID, + URL: row.URL, + StartTime: row.StartTime, + Success: row.Success, + }) + } + + return results, nil +} + // --- Helpers --- func getSingleColumnResultToSlice(ctx context.Context, q *bigquery.Query) ([]string, error) { diff --git a/pkg/api/componentreadiness/dataprovider/interface.go b/pkg/api/componentreadiness/dataprovider/interface.go index 37b40f91e..7f368e2a4 100644 --- a/pkg/api/componentreadiness/dataprovider/interface.go +++ b/pkg/api/componentreadiness/dataprovider/interface.go @@ -66,12 +66,29 @@ type JobQuerier interface { LookupJobVariants(ctx context.Context, jobName string) (map[string]string, error) } +// SpotCheckQuerier fetches job-level pass/fail data for spot-check analysis. +type SpotCheckQuerier interface { + // QuerySpotCheckJobRuns returns aggregated pass/fail per spot-check group, + // grouped by variant columns. Queries the jobs table, not junit. + QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + start, end time.Time) ([]SpotCheckGroup, error) + + // QuerySpotCheckJobRunDetails returns individual job runs for a specific + // spot-check group, used for test details drill-down. + QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, + allJobVariants crtest.JobVariants, + variants map[string]string, + start, end time.Time) ([]JobRunDetail, error) +} + // DataProvider combines all query capabilities needed by Component Readiness. type DataProvider interface { TestStatusQuerier TestDetailsQuerier MetadataQuerier JobQuerier + SpotCheckQuerier // Cache returns the cache implementation for storing/retrieving computed results. Cache() cache.Cache @@ -85,3 +102,21 @@ type JobRunStats struct { SuccessfulRuns int `json:"successful_runs"` PassRate float64 `json:"pass_rate"` } + +// SpotCheckGroup contains aggregated pass/fail for a set of spot-check jobs +// sharing the same variant column values. +type SpotCheckGroup struct { + Variants map[string]string `json:"variants"` + TotalRuns int `json:"total_runs"` + SuccessfulRuns int `json:"successful_runs"` + JobNames []string `json:"job_names"` +} + +// JobRunDetail contains data for a single job run, used in test details drill-down. +type JobRunDetail struct { + JobName string `json:"job_name"` + RunID string `json:"run_id"` + URL string `json:"url"` + StartTime time.Time `json:"start_time"` + Success bool `json:"success"` +} diff --git a/pkg/api/componentreadiness/dataprovider/postgres/provider.go b/pkg/api/componentreadiness/dataprovider/postgres/provider.go index 206f0dac4..508b261ac 100644 --- a/pkg/api/componentreadiness/dataprovider/postgres/provider.go +++ b/pkg/api/componentreadiness/dataprovider/postgres/provider.go @@ -796,3 +796,16 @@ func (p *PostgresProvider) LookupJobVariants(ctx context.Context, jobName string } return parseVariants(row.Variants), nil } + +// --- SpotCheckQuerier --- +// Postgres does not support spot-check queries; these are BigQuery-only. + +func (p *PostgresProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, + _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + return nil, nil +} + +func (p *PostgresProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, + _ crtest.JobVariants, _ map[string]string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + return nil, nil +} diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go new file mode 100644 index 000000000..c9df86156 --- /dev/null +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -0,0 +1,283 @@ +package spotcheckjobs + +import ( + "context" + "fmt" + "strings" + "sync" + + "github.com/openshift/sippy/pkg/api/componentreadiness/dataprovider" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" + log "github.com/sirupsen/logrus" +) + +var _ middleware.Middleware = &SpotCheckJobs{} + +// spotCheckMapping defines the hardcoded component/capability for spot-check jobs +// until SpotCheckComponent/SpotCheckCapability variants exist in the variant registry. +type spotCheckMapping struct { + substrings []string + component string + capability string +} + +var spotCheckMappings = []spotCheckMapping{ + {substrings: []string{"-cpu-partitioning"}, component: "Node", capability: "CPU Partitioning"}, + {substrings: []string{"-etcd-scaling"}, component: "etcd", capability: "Scaling"}, +} + +func NewSpotCheckJobsMiddleware( + provider dataprovider.DataProvider, + reqOptions reqopts.RequestOptions, +) *SpotCheckJobs { + return &SpotCheckJobs{ + dataProvider: provider, + reqOptions: reqOptions, + log: log.WithField("middleware", "SpotCheckJobs"), + } +} + +type SpotCheckJobs struct { + dataProvider dataprovider.DataProvider + reqOptions reqopts.RequestOptions + log log.FieldLogger + + // sampleJobDetails is populated during QueryTestDetails and consumed by PreTestDetailsAnalysis. + sampleJobDetails map[string][]dataprovider.JobRunDetail + sampleJobDetailsMutex sync.Mutex +} + +func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, + allJobVariants crtest.JobVariants, + _, sampleStatusCh chan map[string]crstatus.TestStatus, errCh chan error) { + + if s.reqOptions.SpotCheckSample == nil { + return + } + + wg.Add(1) + go func() { + defer wg.Done() + select { + case <-ctx.Done(): + s.log.Info("context canceled during spot check query") + return + default: + } + + groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, + s.reqOptions, allJobVariants, + s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + if err != nil { + errCh <- fmt.Errorf("spot check query failed: %w", err) + return + } + + sampleStatus := map[string]crstatus.TestStatus{} + for _, group := range groups { + component, capability := s.resolveComponentCapability(group) + if component == "" { + continue + } + + testKey := crtest.KeyWithVariants{ + TestID: syntheticTestID(component, capability), + Variants: group.Variants, + } + keyStr := testKey.KeyOrDie() + + atLeastOnePass := group.SuccessfulRuns >= 1 + successCount := 0 + if atLeastOnePass { + successCount = 1 + } + + sampleStatus[keyStr] = crstatus.TestStatus{ + TestName: syntheticTestName(component, capability), + Component: component, + Capabilities: []string{capability}, + Variants: variantMapToSlice(group.Variants), + Count: crtest.Count{ + TotalCount: 1, + SuccessCount: successCount, + }, + } + } + + s.log.Infof("injecting %d spot-check synthetic test results", len(sampleStatus)) + sampleStatusCh <- sampleStatus + }() +} + +func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup, + errCh chan error, allJobVariants crtest.JobVariants) { + + if s.reqOptions.SpotCheckSample == nil { + return + } + + if len(s.reqOptions.TestIDOptions) == 0 || !isSpotCheckTestID(s.reqOptions.TestIDOptions[0].TestID) { + return + } + + wg.Add(1) + go func() { + defer wg.Done() + select { + case <-ctx.Done(): + return + default: + } + + testIDOpt := s.reqOptions.TestIDOptions[0] + details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, + s.reqOptions, allJobVariants, + testIDOpt.RequestedVariants, + s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + if err != nil { + errCh <- fmt.Errorf("spot check details query failed: %w", err) + return + } + + testKey := crtest.KeyWithVariants{ + TestID: testIDOpt.TestID, + Variants: testIDOpt.RequestedVariants, + } + + s.sampleJobDetailsMutex.Lock() + defer s.sampleJobDetailsMutex.Unlock() + if s.sampleJobDetails == nil { + s.sampleJobDetails = map[string][]dataprovider.JobRunDetail{} + } + s.sampleJobDetails[testKey.KeyOrDie()] = details + s.log.Infof("loaded %d spot-check job run details", len(details)) + }() +} + +func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, + testStats *testdetails.TestComparison) error { + + if !isSpotCheckTestID(testKey.TestID) { + return nil + } + + sampleDays := int(s.reqOptions.SpotCheckSample.End.Sub(s.reqOptions.SpotCheckSample.Start).Hours() / 24) + + if testStats.SampleStats.SuccessCount > 0 { + testStats.ReportStatus = crtest.NotSignificant + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Spot-check job passed at least once in the %d-day sample window", sampleDays)) + } else { + testStats.ReportStatus = crtest.ExtremeRegression + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Spot-check job did not pass in the %d-day sample window (%d runs, 0 successes)", + sampleDays, testStats.SampleStats.Total())) + } + + testStats.Comparison = crtest.SpotCheck + testStats.AnalysisComplete = true + testStats.BaseStats = nil + + return nil +} + +func (s *SpotCheckJobs) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, + status *crstatus.TestJobRunStatuses) error { + + if !isSpotCheckTestID(testKey.TestID) { + return nil + } + + s.sampleJobDetailsMutex.Lock() + details := s.sampleJobDetails[testKey.KeyOrDie()] + s.sampleJobDetailsMutex.Unlock() + + if status.SampleStatus == nil { + status.SampleStatus = map[string][]crstatus.TestJobRunRows{} + } + + for _, run := range details { + successCount := 0 + if run.Success { + successCount = 1 + } + row := crstatus.TestJobRunRows{ + TestKey: testKey, + TestKeyStr: testKey.KeyOrDie(), + TestName: syntheticTestNameFromID(testKey.TestID), + ProwJob: run.JobName, + ProwJobRunID: run.RunID, + ProwJobURL: run.URL, + StartTime: run.StartTime, + Count: crtest.Count{ + TotalCount: 1, + SuccessCount: successCount, + }, + } + status.SampleStatus[run.JobName] = append(status.SampleStatus[run.JobName], row) + } + + return nil +} + +// resolveComponentCapability maps a spot-check group to its component/capability +// using hardcoded patterns based on job names. This will be replaced by +// SpotCheckComponent/SpotCheckCapability variants once the variant registry is updated. +func (s *SpotCheckJobs) resolveComponentCapability(group dataprovider.SpotCheckGroup) (string, string) { + for _, jobName := range group.JobNames { + lower := strings.ToLower(jobName) + for _, m := range spotCheckMappings { + allMatch := true + for _, sub := range m.substrings { + if !strings.Contains(lower, sub) { + allMatch = false + break + } + } + if allMatch { + return m.component, m.capability + } + } + } + return "", "" +} + +func isSpotCheckTestID(testID string) bool { + return strings.HasPrefix(testID, "spotcheck:") +} + +func syntheticTestID(component, capability string) string { + return fmt.Sprintf("spotcheck:%s:%s", + strings.ToLower(component), + strings.ToLower(strings.ReplaceAll(capability, " ", "-"))) +} + +func syntheticTestName(component, capability string) string { + return fmt.Sprintf("[spot-check] %s / %s must pass at least once per sample window", + component, capability) +} + +func syntheticTestNameFromID(testID string) string { + parts := strings.SplitN(testID, ":", 3) + if len(parts) == 3 { + return fmt.Sprintf("[spot-check] %s / %s must pass at least once per sample window", + parts[1], strings.ReplaceAll(parts[2], "-", " ")) + } + return testID +} + +func variantMapToSlice(m map[string]string) []string { + result := make([]string, 0, len(m)) + for k, v := range m { + result = append(result, k+":"+v) + } + return result +} diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index d63465a41..56ffad83a 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -55,6 +55,15 @@ func ParseComponentReportRequest( viewWarnings := api.ValidateVariants(allJobVariants, opts.VariantOption.IncludeVariants, " from view") warnings = append(warnings, viewWarnings...) } + + if view.SpotCheckSample != nil { + resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", *view.SpotCheckSample, crTimeRoundingFactor) + if resolveErr != nil { + err = resolveErr + return + } + opts.SpotCheckSample = &resolved + } } // Parse URL parameters - these override view defaults if view was provided diff --git a/pkg/apis/api/componentreport/crtest/types.go b/pkg/apis/api/componentreport/crtest/types.go index 95900a852..15d67a98e 100644 --- a/pkg/apis/api/componentreport/crtest/types.go +++ b/pkg/apis/api/componentreport/crtest/types.go @@ -35,6 +35,7 @@ type Comparison string const ( PassRate Comparison = "pass_rate" FisherExact Comparison = "fisher_exact" + SpotCheck Comparison = "spot_check" ) const ( diff --git a/pkg/apis/api/componentreport/crview/types.go b/pkg/apis/api/componentreport/crview/types.go index ed2e2fa07..34741b972 100644 --- a/pkg/apis/api/componentreport/crview/types.go +++ b/pkg/apis/api/componentreport/crview/types.go @@ -13,6 +13,11 @@ type View struct { VariantOptions reqopts.Variants `json:"variant_options" yaml:"variant_options"` AdvancedOptions reqopts.Advanced `json:"advanced_options" yaml:"advanced_options"` + // SpotCheckSample defines the sample window for spot-check job analysis. + // Spot-check jobs must pass at least once in this window (typically 30 days). + // If nil, spot-check analysis is disabled for this view. + SpotCheckSample *reqopts.RelativeRelease `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` + Metrics Metrics `json:"metrics" yaml:"metrics"` RegressionTracking RegressionTracking `json:"regression_tracking" yaml:"regression_tracking"` AutomateJira AutomateJira `json:"automate_jira" yaml:"automate_jira"` diff --git a/pkg/apis/api/componentreport/reqopts/types.go b/pkg/apis/api/componentreport/reqopts/types.go index bba5202d7..5a43297a5 100644 --- a/pkg/apis/api/componentreport/reqopts/types.go +++ b/pkg/apis/api/componentreport/reqopts/types.go @@ -19,6 +19,10 @@ type RequestOptions struct { CacheOption cache.RequestOptions TestFilters TestIDOptions []TestIdentification + // SpotCheckSample is the resolved sample window for spot-check job analysis. + // Only set when the view defines spot_check_sample. These jobs must pass at least + // once in this window (typically 30 days, longer than the normal 7-day test sample). + SpotCheckSample *Release `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` // ViewName is the name of the view used for this request, if any. // When generating test details URLs, if a view is present, we include just the view parameter // plus test-specific overrides, rather than expanding all view parameters into the URL. diff --git a/pkg/apis/api/componentreport/testdetails/types.go b/pkg/apis/api/componentreport/testdetails/types.go index c900e3ddb..37e89c07a 100644 --- a/pkg/apis/api/componentreport/testdetails/types.go +++ b/pkg/apis/api/componentreport/testdetails/types.go @@ -79,6 +79,10 @@ type TestComparison struct { // LastFailure is the last time the regressed test failed. LastFailure *time.Time `json:"last_failure"` + // AnalysisComplete indicates middleware has already determined the report status. + // When true, assessComponentStatus skips its analysis (Fisher's, pass-rate, etc.). + AnalysisComplete bool `json:"-"` + // Regression is populated with data on when we first detected this regression. If unset it implies // the regression tracker has not yet run to find it, or you're using report params/a view without regression tracking. Regression *models.TestRegression `json:"regression,omitempty"` diff --git a/pkg/bigquery/bqlabel/labels.go b/pkg/bigquery/bqlabel/labels.go index f9d4c0426..34791400e 100644 --- a/pkg/bigquery/bqlabel/labels.go +++ b/pkg/bigquery/bqlabel/labels.go @@ -73,6 +73,8 @@ const ( CRJunitSample QueryValue = "component-readiness-junit-sample" CRJunitFallback QueryValue = "component-readiness-junit-fallback" CRViewJobs QueryValue = "component-readiness-view-jobs" + CRSpotCheck QueryValue = "component-readiness-spot-check" + CRSpotCheckDetails QueryValue = "component-readiness-spot-check-details" TDJunitBase QueryValue = "test-details-junit-base" TDJunitSample QueryValue = "test-details-junit-sample" DisruptionDelta QueryValue = "disruption-delta" diff --git a/sippy-ng/src/component_readiness/CompReadyTestDetailRow.js b/sippy-ng/src/component_readiness/CompReadyTestDetailRow.js index c32065095..50eddfc12 100644 --- a/sippy-ng/src/component_readiness/CompReadyTestDetailRow.js +++ b/sippy-ng/src/component_readiness/CompReadyTestDetailRow.js @@ -35,6 +35,7 @@ export default function CompReadyTestDetailRow(props) { const { element, idx, + isSpotCheck, showOnlyFailures, searchJobArtifacts, searchJobRunIds, @@ -198,27 +199,29 @@ export default function CompReadyTestDetailRow(props) { return ( - - - {element.base_job_name ? ( - - {element.base_job_name} - {searchJobArtifacts && - getSelectingCheckBox(element.base_job_run_stats)} - - ) : ( - '(No basis equivalent)' - )} - - + {!isSpotCheck && ( + + + {element.base_job_name ? ( + + {element.base_job_name} + {searchJobArtifacts && + getSelectingCheckBox(element.base_job_run_stats)} + + ) : ( + '(No basis equivalent)' + )} + + + )} {element.sample_job_name ? ( @@ -234,20 +237,24 @@ export default function CompReadyTestDetailRow(props) { - - {element.base_job_name ? infoCell(element.base_stats) : ''} - - {testJobDetailCell(element, 'base')} - + {!isSpotCheck && ( + + {element.base_job_name ? infoCell(element.base_stats) : ''} + + )} + {!isSpotCheck && testJobDetailCell(element, 'base')} + {!isSpotCheck && } {element.sample_job_name ? infoCell(element.sample_stats) : ''} {testJobDetailCell(element, 'sample')} - - - {element.significant ? 'True' : 'False'} - - + {!isSpotCheck && ( + + + {element.significant ? 'True' : 'False'} + + + )} ) @@ -256,6 +263,7 @@ export default function CompReadyTestDetailRow(props) { CompReadyTestDetailRow.propTypes = { element: PropTypes.object.isRequired, idx: PropTypes.number.isRequired, + isSpotCheck: PropTypes.bool, showOnlyFailures: PropTypes.bool.isRequired, searchJobArtifacts: PropTypes.bool.isRequired, searchJobRunIds: PropTypes.object.isRequired, diff --git a/sippy-ng/src/component_readiness/CompReadyTestPanel.js b/sippy-ng/src/component_readiness/CompReadyTestPanel.js index a665a1572..c06835366 100644 --- a/sippy-ng/src/component_readiness/CompReadyTestPanel.js +++ b/sippy-ng/src/component_readiness/CompReadyTestPanel.js @@ -24,8 +24,16 @@ import PropTypes from 'prop-types' import React, { Fragment, useContext } from 'react' export default function CompReadyTestPanel(props) { - const { data, versions, loadedParams, testName, environment, component } = - props + const { + data, + comparison, + versions, + loadedParams, + testName, + environment, + component, + } = props + const isSpotCheck = comparison === 'spot_check' const classes = useContext(ComponentReadinessStyleContext) const significanceTitle = `Test results for individual Prow Jobs may not be statistically @@ -224,7 +232,7 @@ export default function CompReadyTestPanel(props) { return ( - {data.base_stats && ( + {!isSpotCheck && data.base_stats && ( {printParamsAndStats( 'Basis (historical)', @@ -237,9 +245,9 @@ export default function CompReadyTestPanel(props) { )} {data.sample_stats && data.sample_stats.Start && data.sample_stats.End && ( - + {printParamsAndStats( - 'Sample (being evaluated)', + isSpotCheck ? 'Spot-Check Window' : 'Sample (being evaluated)', data.sample_stats, data.sample_stats.Start.toString(), data.sample_stats.End.toString(), @@ -288,16 +296,17 @@ export default function CompReadyTestPanel(props) { - {tableCell('Basis Job', 0)} - {tableCell('Basis Runs', 1)} - {tableCell('', 2)} - {tableCell('Sample Job', 3)} - {tableCell('Sample Runs', 4)} - {tableTooltipCell( - 'Statistically Significant', - 5, - significanceTitle - )} + {!isSpotCheck && tableCell('Basis Job', 0)} + {!isSpotCheck && tableCell('Basis Runs', 1)} + {!isSpotCheck && tableCell('', 2)} + {tableCell(isSpotCheck ? 'Job' : 'Sample Job', 3)} + {tableCell(isSpotCheck ? 'Runs' : 'Sample Runs', 4)} + {!isSpotCheck && + tableTooltipCell( + 'Statistically Significant', + 5, + significanceTitle + )} @@ -320,6 +329,7 @@ export default function CompReadyTestPanel(props) { key={idx} element={element} idx={idx} + isSpotCheck={isSpotCheck} showOnlyFailures={showOnlyFailures} searchJobArtifacts={searchJobArtifacts} searchJobRunIds={searchJobRunIds} @@ -402,6 +412,7 @@ export default function CompReadyTestPanel(props) { CompReadyTestPanel.propTypes = { data: PropTypes.object.isRequired, + comparison: PropTypes.string, versions: PropTypes.object.isRequired, loadedParams: PropTypes.object.isRequired, testName: PropTypes.string.isRequired, diff --git a/sippy-ng/src/component_readiness/TestDetailsReport.js b/sippy-ng/src/component_readiness/TestDetailsReport.js index 8265b3b01..2dad0b711 100644 --- a/sippy-ng/src/component_readiness/TestDetailsReport.js +++ b/sippy-ng/src/component_readiness/TestDetailsReport.js @@ -365,6 +365,8 @@ export default function TestDetailsReport(props) { 0, accessibilityModeOn ) + const isSpotCheck = data.analyses[0].comparison === 'spot_check' + const significanceTitle = `Test results for individual Prow Jobs may not be statistically significant, but when taken in aggregate, there may be a statistically significant difference compared to the historical basis @@ -621,7 +623,7 @@ View the [test details report|${document.location.href}] for additional context. Environment: {environment} - {isBaseOverride && ( + {isBaseOverride && !isSpotCheck && ( {baseRelease} Override: Earlier release had a higher threshold @@ -667,6 +669,7 @@ View the [test details report|${document.location.href}] for additional context. Date: Thu, 14 May 2026 14:55:02 -0300 Subject: [PATCH 03/37] Results showing in main report but not test details yet --- .../dataprovider/bigquery/provider.go | 2 +- .../middleware/spotcheckjobs/spotcheckjobs.go | 66 ++++++++++--------- pkg/api/componentreadiness/test_details.go | 56 +++++++++------- .../regressioncacheloader.go | 9 +++ 4 files changed, 77 insertions(+), 56 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 4fb1d532a..207125d97 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -537,7 +537,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO jobs.prowjob_job_name AS job_name, jobs.prowjob_build_id AS run_id, jobs.prowjob_url AS url, - jobs.prowjob_start AS start_time, + TIMESTAMP(jobs.prowjob_start) AS start_time, (jobs.prowjob_state = 'success') AS success FROM %s.jobs jobs %s diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index c9df86156..826fc2016 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -120,42 +120,44 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup return } - if len(s.reqOptions.TestIDOptions) == 0 || !isSpotCheckTestID(s.reqOptions.TestIDOptions[0].TestID) { - return - } - - wg.Add(1) - go func() { - defer wg.Done() - select { - case <-ctx.Done(): - return - default: + for _, testIDOpt := range s.reqOptions.TestIDOptions { + if !isSpotCheckTestID(testIDOpt.TestID) { + continue } - testIDOpt := s.reqOptions.TestIDOptions[0] - details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, - s.reqOptions, allJobVariants, - testIDOpt.RequestedVariants, - s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) - if err != nil { - errCh <- fmt.Errorf("spot check details query failed: %w", err) - return - } + opt := testIDOpt + wg.Add(1) + go func() { + defer wg.Done() + select { + case <-ctx.Done(): + return + default: + } - testKey := crtest.KeyWithVariants{ - TestID: testIDOpt.TestID, - Variants: testIDOpt.RequestedVariants, - } + details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, + s.reqOptions, allJobVariants, + opt.RequestedVariants, + s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + if err != nil { + errCh <- fmt.Errorf("spot check details query failed: %w", err) + return + } - s.sampleJobDetailsMutex.Lock() - defer s.sampleJobDetailsMutex.Unlock() - if s.sampleJobDetails == nil { - s.sampleJobDetails = map[string][]dataprovider.JobRunDetail{} - } - s.sampleJobDetails[testKey.KeyOrDie()] = details - s.log.Infof("loaded %d spot-check job run details", len(details)) - }() + testKey := crtest.KeyWithVariants{ + TestID: opt.TestID, + Variants: opt.RequestedVariants, + } + + s.sampleJobDetailsMutex.Lock() + defer s.sampleJobDetailsMutex.Unlock() + if s.sampleJobDetails == nil { + s.sampleJobDetails = map[string][]dataprovider.JobRunDetail{} + } + s.sampleJobDetails[testKey.KeyOrDie()] = details + s.log.Infof("loaded %d spot-check job run details for %s", len(details), opt.TestID) + }() + } } func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, diff --git a/pkg/api/componentreadiness/test_details.go b/pkg/api/componentreadiness/test_details.go index 26c900c53..ef6db5ff5 100644 --- a/pkg/api/componentreadiness/test_details.go +++ b/pkg/api/componentreadiness/test_details.go @@ -7,6 +7,7 @@ import ( "os" "slices" "sort" + "strings" "sync" "time" @@ -161,18 +162,24 @@ func (c *ComponentReportGenerator) GenerateTestDetailsReportMultiTest(ctx contex Variants: tOpt.RequestedVariants, } testKeyStr := testKey.KeyOrDie() - if statuses, ok := testKeyTestJobRunStatuses[testKeyStr]; ok { - report, generateReportErrs := c.GenerateDetailsReportForTest(ctx, tOpt, statuses, false) - if len(generateReportErrs) > 0 { - errs = append(errs, generateReportErrs...) - continue + statuses, ok := testKeyTestJobRunStatuses[testKeyStr] + if !ok { + // Spot-check synthetic tests won't appear in junit query results. + // Create an empty status so GenerateDetailsReportForTest can still run; + // the middleware will populate it via PreTestDetailsAnalysis. + statuses = crstatus.TestJobRunStatuses{ + BaseStatus: map[string][]crstatus.TestJobRunRows{}, + BaseOverrideStatus: map[string][]crstatus.TestJobRunRows{}, + SampleStatus: map[string][]crstatus.TestJobRunRows{}, + GeneratedAt: allTestsJobRunStatuses.GeneratedAt, } - reports = append(reports, report) - } else { - logrus.Errorf("missing test key in results: %v", testKeyStr) - } - + report, generateReportErrs := c.GenerateDetailsReportForTest(ctx, tOpt, statuses, false) + if len(generateReportErrs) > 0 { + errs = append(errs, generateReportErrs...) + continue + } + reports = append(reports, report) } return reports, errs } @@ -188,11 +195,14 @@ func (c *ComponentReportGenerator) GenerateDetailsReportForTest( if testIDOption.TestID == "" { return testdetails.Report{}, []error{fmt.Errorf("test_id has to be defined for test details")} } - for _, v := range c.ReqOptions.VariantOption.DBGroupBy.List() { - if _, ok := testIDOption.RequestedVariants[v]; !ok { - return testdetails.Report{}, []error{ - fmt.Errorf("all dbGroupBy variants have to be defined for test details: %s is missing in %v", - v, testIDOption.RequestedVariants), + isSpotCheck := strings.HasPrefix(testIDOption.TestID, "spotcheck:") + if !isSpotCheck { + for _, v := range c.ReqOptions.VariantOption.DBGroupBy.List() { + if _, ok := testIDOption.RequestedVariants[v]; !ok { + return testdetails.Report{}, []error{ + fmt.Errorf("all dbGroupBy variants have to be defined for test details: %s is missing in %v", + v, testIDOption.RequestedVariants), + } } } } @@ -220,18 +230,18 @@ func (c *ComponentReportGenerator) GenerateDetailsReportForTest( // to a circular dep. This is an unfortunate compromise in the middleware goal I didn't have time to unwind. // For now, the middleware does the querying for test details, and passes the override status out // by adding it to componentJobRunTestReportStatus.BaseOverrideStatus. + testKey := crtest.KeyWithVariants{ + TestID: testIDOption.TestID, + Variants: testIDOption.RequestedVariants, + } + if err := c.middlewares.PreTestDetailsAnalysis(testKey, &componentJobRunTestReportStatus); err != nil { + return testdetails.Report{}, []error{err} + } + var baseOverrideReport *testdetails.Report if testIDOption.BaseOverrideRelease != "" && testIDOption.BaseOverrideRelease != c.ReqOptions.BaseRelease.Name { - testKey := crtest.KeyWithVariants{ - TestID: testIDOption.TestID, - Variants: testIDOption.RequestedVariants, - } - if err := c.middlewares.PreTestDetailsAnalysis(testKey, &componentJobRunTestReportStatus); err != nil { - return testdetails.Report{}, []error{err} - } - start, end, err := utils.FindStartEndTimesForRelease(timeRanges, testIDOption.BaseOverrideRelease) if err != nil { return testdetails.Report{}, []error{err} diff --git a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go index 43e5f56f5..fc7fea7d3 100644 --- a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go +++ b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go @@ -505,6 +505,15 @@ func (l *RegressionCacheLoader) buildGenerator( TestFilters: view.TestFilters, } + if view.SpotCheckSample != nil { + resolved, err := utils.GetViewReleaseOptions( + l.releases, "spot_check", *view.SpotCheckSample, cacheOpts.CRTimeRoundingFactor) + if err != nil { + return nil, err + } + reqOpts.SpotCheckSample = &resolved + } + generator := componentreadiness.NewComponentReportGenerator( bqprovider.NewBigQueryProvider(l.bqClient), reqOpts, l.dbc, From 569ab9965b971ab0c1e6e2b74dcfe37c4c4469d3 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 15 May 2026 10:28:33 -0300 Subject: [PATCH 04/37] Fix test details report pages --- pkg/api/componentreadiness/test_details.go | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/pkg/api/componentreadiness/test_details.go b/pkg/api/componentreadiness/test_details.go index ef6db5ff5..1262e2fb0 100644 --- a/pkg/api/componentreadiness/test_details.go +++ b/pkg/api/componentreadiness/test_details.go @@ -215,6 +215,15 @@ func (c *ComponentReportGenerator) GenerateDetailsReportForTest( now := time.Now() componentJobRunTestReportStatus.GeneratedAt = &now + // Let middleware inject data (e.g. spot-check job runs) before generating reports. + testKey := crtest.KeyWithVariants{ + TestID: testIDOption.TestID, + Variants: testIDOption.RequestedVariants, + } + if err := c.middlewares.PreTestDetailsAnalysis(testKey, &componentJobRunTestReportStatus); err != nil { + return testdetails.Report{}, []error{err} + } + // Generate the report for the main release that was originally requested: report := c.internalGenerateTestDetailsReport( c.ReqOptions.BaseRelease.Name, @@ -230,14 +239,6 @@ func (c *ComponentReportGenerator) GenerateDetailsReportForTest( // to a circular dep. This is an unfortunate compromise in the middleware goal I didn't have time to unwind. // For now, the middleware does the querying for test details, and passes the override status out // by adding it to componentJobRunTestReportStatus.BaseOverrideStatus. - testKey := crtest.KeyWithVariants{ - TestID: testIDOption.TestID, - Variants: testIDOption.RequestedVariants, - } - if err := c.middlewares.PreTestDetailsAnalysis(testKey, &componentJobRunTestReportStatus); err != nil { - return testdetails.Report{}, []error{err} - } - var baseOverrideReport *testdetails.Report if testIDOption.BaseOverrideRelease != "" && testIDOption.BaseOverrideRelease != c.ReqOptions.BaseRelease.Name { From c6d37a9588e4b108ddef17c43a401ac2b61290ee Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 15 May 2026 10:31:23 -0300 Subject: [PATCH 05/37] Simplify spot check struct to prevent mismatched releases and allow future config --- config/views.yaml | 1 - pkg/api/componentreadiness/utils/queryparamparser.go | 7 ++++++- pkg/apis/api/componentreport/crview/types.go | 10 +++++++++- .../regressioncacheloader/regressioncacheloader.go | 7 ++++++- 4 files changed, 21 insertions(+), 4 deletions(-) diff --git a/config/views.yaml b/config/views.yaml index b15414b95..8ba683cd1 100755 --- a/config/views.yaml +++ b/config/views.yaml @@ -10,7 +10,6 @@ component_readiness: relative_start: now-7d relative_end: now spot_check_sample: - release: "5.0" relative_start: now-30d relative_end: now variant_options: diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index 56ffad83a..6f70c6968 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -57,7 +57,12 @@ func ParseComponentReportRequest( } if view.SpotCheckSample != nil { - resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", *view.SpotCheckSample, crTimeRoundingFactor) + spotCheckRelative := reqopts.RelativeRelease{ + Release: reqopts.Release{Name: view.SampleRelease.Name}, + RelativeStart: view.SpotCheckSample.RelativeStart, + RelativeEnd: view.SpotCheckSample.RelativeEnd, + } + resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", spotCheckRelative, crTimeRoundingFactor) if resolveErr != nil { err = resolveErr return diff --git a/pkg/apis/api/componentreport/crview/types.go b/pkg/apis/api/componentreport/crview/types.go index 34741b972..4eb1bbe8e 100644 --- a/pkg/apis/api/componentreport/crview/types.go +++ b/pkg/apis/api/componentreport/crview/types.go @@ -16,7 +16,8 @@ type View struct { // SpotCheckSample defines the sample window for spot-check job analysis. // Spot-check jobs must pass at least once in this window (typically 30 days). // If nil, spot-check analysis is disabled for this view. - SpotCheckSample *reqopts.RelativeRelease `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` + // The release is inherited from SampleRelease; only the time window needs to be specified. + SpotCheckSample *SpotCheckWindow `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` Metrics Metrics `json:"metrics" yaml:"metrics"` RegressionTracking RegressionTracking `json:"regression_tracking" yaml:"regression_tracking"` @@ -38,3 +39,10 @@ type PrimeCache struct { type AutomateJira struct { Enabled bool `json:"enabled" yaml:"enabled"` } + +// SpotCheckWindow defines just the time window for spot-check analysis. +// The release is always inherited from SampleRelease. +type SpotCheckWindow struct { + RelativeStart string `json:"relative_start,omitempty" yaml:"relative_start,omitempty"` + RelativeEnd string `json:"relative_end,omitempty" yaml:"relative_end,omitempty"` +} diff --git a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go index fc7fea7d3..ce93a9c2c 100644 --- a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go +++ b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go @@ -506,8 +506,13 @@ func (l *RegressionCacheLoader) buildGenerator( } if view.SpotCheckSample != nil { + spotCheckRelative := reqopts.RelativeRelease{ + Release: reqopts.Release{Name: view.SampleRelease.Name}, + RelativeStart: view.SpotCheckSample.RelativeStart, + RelativeEnd: view.SpotCheckSample.RelativeEnd, + } resolved, err := utils.GetViewReleaseOptions( - l.releases, "spot_check", *view.SpotCheckSample, cacheOpts.CRTimeRoundingFactor) + l.releases, "spot_check", spotCheckRelative, cacheOpts.CRTimeRoundingFactor) if err != nil { return nil, err } From 9bcd7e14a08df7aaaf9715cdcd45db12f0332d84 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 15 May 2026 10:41:13 -0300 Subject: [PATCH 06/37] Clarify synthetic test name a bit --- .../middleware/spotcheckjobs/spotcheckjobs.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 826fc2016..44b9a4c37 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -263,14 +263,14 @@ func syntheticTestID(component, capability string) string { } func syntheticTestName(component, capability string) string { - return fmt.Sprintf("[spot-check] %s / %s must pass at least once per sample window", + return fmt.Sprintf("[spot-check] %s / %s job must pass at least once per sample window", component, capability) } func syntheticTestNameFromID(testID string) string { parts := strings.SplitN(testID, ":", 3) if len(parts) == 3 { - return fmt.Sprintf("[spot-check] %s / %s must pass at least once per sample window", + return fmt.Sprintf("[spot-check] %s / %s job must pass at least once per sample window", parts[1], strings.ReplaceAll(parts[2], "-", " ")) } return testID From fe2222e5290e63b1abd3e09adda172900e67547f Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Tue, 26 May 2026 09:21:19 -0300 Subject: [PATCH 07/37] Add AI generated function comments --- .../dataprovider/bigquery/provider.go | 7 +++++++ .../middleware/spotcheckjobs/spotcheckjobs.go | 16 ++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 207125d97..51443f467 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -376,6 +376,10 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // --- SpotCheckQuerier --- +// QuerySpotCheckJobRuns queries the jobs table (not junit) for rare-tier periodic/release +// jobs, grouping by the requested variant columns (e.g. Platform, Architecture, Network). +// Returns aggregated pass/fail counts per variant group so the middleware can create +// synthetic test results without needing individual test case data. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { @@ -505,6 +509,9 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions return results, nil } +// QuerySpotCheckJobRunDetails returns individual job runs matching the given variant +// filters, used to populate the test details drill-down page for a specific spot-check +// synthetic test. func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 44b9a4c37..f8ae2c147 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -30,6 +30,10 @@ var spotCheckMappings = []spotCheckMapping{ {substrings: []string{"-etcd-scaling"}, component: "etcd", capability: "Scaling"}, } +// NewSpotCheckJobsMiddleware creates middleware that injects synthetic test results for +// "rare" tier jobs (e.g. cpu-partitioning, etcd-scaling) that don't run in the standard +// junit-based test pipeline. These jobs are evaluated on a simple pass/fail basis: +// at least one successful run in the sample window means healthy. func NewSpotCheckJobsMiddleware( provider dataprovider.DataProvider, reqOptions reqopts.RequestOptions, @@ -51,6 +55,9 @@ type SpotCheckJobs struct { sampleJobDetailsMutex sync.Mutex } +// Query fetches aggregated spot-check job results from BigQuery, creates synthetic +// test statuses (one per variant group), and injects them into the sample status channel. +// Each synthetic test uses a binary pass/fail: >=1 successful run = pass. func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, allJobVariants crtest.JobVariants, _, sampleStatusCh chan map[string]crstatus.TestStatus, errCh chan error) { @@ -113,6 +120,8 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, }() } +// QueryTestDetails fetches individual job run details for spot-check synthetic tests, +// storing them for later use by PreTestDetailsAnalysis to populate the drill-down view. func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup, errCh chan error, allJobVariants crtest.JobVariants) { @@ -160,6 +169,10 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup } } +// PreAnalysis overrides the normal fisher-exact statistical comparison for spot-check +// tests. Instead it applies a simple heuristic: any successful run in the sample window +// means the job is healthy (NotSignificant), zero successes means ExtremeRegression. +// Marks analysis as complete so the standard pipeline skips further processing. func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error { @@ -191,6 +204,9 @@ func (s *SpotCheckJobs) PostAnalysis(_ crtest.Identification, _ *testdetails.Tes return nil } +// PreTestDetailsAnalysis populates the test details drill-down view for spot-check tests +// by converting the cached job run details into TestJobRunRows for the sample side. +// There is no base side since spot-check tests have no historical baseline comparison. func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, status *crstatus.TestJobRunStatuses) error { From 952a40fc390541fa065b5827b501ca939640b5f6 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Tue, 26 May 2026 10:17:50 -0300 Subject: [PATCH 08/37] Refactor Analysis to an impl on component readiness middlewares --- .../componentreadiness/component_report.go | 185 ++---------------- .../component_report_test.go | 15 +- .../alltestspassrate/alltestspassrate.go | 51 +++++ .../middleware/analysis/passrate.go | 39 ++++ .../middleware/fishersexact/fishersexact.go | 153 +++++++++++++++ .../middleware/interface.go | 6 + .../middleware/linkinjector/linkinjector.go | 4 + pkg/api/componentreadiness/middleware/list.go | 13 ++ .../newtestpassrate/newtestpassrate.go | 56 ++++++ .../regressionallowances.go | 4 + .../regressiontracker/regressiontracker.go | 4 + .../releasefallback/releasefallback.go | 4 +- .../middleware/spotcheckjobs/spotcheckjobs.go | 21 +- pkg/api/componentreadiness/test_details.go | 4 +- .../api/componentreport/testdetails/types.go | 4 - 15 files changed, 367 insertions(+), 196 deletions(-) create mode 100644 pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate.go create mode 100644 pkg/api/componentreadiness/middleware/analysis/passrate.go create mode 100644 pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go create mode 100644 pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate.go diff --git a/pkg/api/componentreadiness/component_report.go b/pkg/api/componentreadiness/component_report.go index e1c30505e..d04329d68 100644 --- a/pkg/api/componentreadiness/component_report.go +++ b/pkg/api/componentreadiness/component_report.go @@ -3,7 +3,6 @@ package componentreadiness import ( "context" "encoding/json" - "fmt" "maps" "os" "reflect" @@ -12,9 +11,10 @@ import ( "sync" "time" - "github.com/apache/thrift/lib/go/thrift" - fischer "github.com/glycerine/golang-fisher-exact" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/alltestspassrate" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/fishersexact" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/linkinjector" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/newtestpassrate" regressionallowances2 "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/regressionallowances" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/regressiontracker" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/spotcheckjobs" @@ -38,7 +38,6 @@ import ( ) const ( - explanationNoRegression = "No significant regressions found" ComponentReportCacheKeyPrefix = "ComponentReport~" TestDetailsReportCacheKeyPrefix = "TestDetailsReport~" ) @@ -294,6 +293,11 @@ func (c *ComponentReportGenerator) initializeMiddleware() { } c.middlewares = append(c.middlewares, regressionallowances2.NewRegressionAllowancesMiddleware(c.ReqOptions, c.releaseConfigs)) + // Analysis middleware ordered by priority — first responder wins. + c.middlewares = append(c.middlewares, newtestpassrate.NewNewTestPassRateMiddleware(c.ReqOptions)) + c.middlewares = append(c.middlewares, alltestspassrate.NewAllTestsPassRateMiddleware(c.ReqOptions)) + c.middlewares = append(c.middlewares, fishersexact.NewFishersExactMiddleware(c.ReqOptions)) + // Initialize LinkInjector middleware linkInjector := linkinjector.NewLinkInjectorMiddleware(c.ReqOptions, c.baseURL) c.middlewares = append(c.middlewares, linkInjector) @@ -651,7 +655,9 @@ func (c *ComponentReportGenerator) generateComponentTestReport(basisStatusMap, s return crtype.ComponentReport{}, err } - c.assessComponentStatus(&cellReport, log.NewEntry(log.New())) + if _, err := c.middlewares.Analyze(testKey, &cellReport); err != nil { + return crtype.ComponentReport{}, err + } if lastFailure := sampleStatus.LastFailure; !lastFailure.IsZero() { cellReport.LastFailure = &lastFailure // it's a copy, for pointer hygiene } @@ -754,175 +760,6 @@ func buildReport(sortedRows []crtest.RowIdentification, sortedColumns []crtest.C return regressionRows, nil } -func getRegressionStatus(basisPassPercentage, samplePassPercentage float64) crtest.Status { - if (basisPassPercentage - samplePassPercentage) > 0.15 { - return crtest.ExtremeRegression - } - - return crtest.SignificantRegression -} - -// TODO: this will eventually become the analyze step on a Middleware, or possibly a separate -// set of objects relating to analysis, as there's not a lot of overlap between the analyzers -// (fishers, pass rate, bayes (future)) and the middlewares (fallback, intentional regressions, -// cross variant compare, rarely run jobs, etc.) -func (c *ComponentReportGenerator) assessComponentStatus(testStats *testdetails.TestComparison, logger *log.Entry) { - if testStats.AnalysisComplete { - return - } - // Catch unset required confidence, typically unit tests - opts := c.ReqOptions.AdvancedOption - if testStats.RequiredConfidence == 0 { - testStats.RequiredConfidence = opts.Confidence - } - - if (testStats.BaseStats == nil || testStats.BaseStats.Total() == 0) && opts.PassRateRequiredNewTests > 0 { - // If we have no base stats, fall back to a raw pass rate comparison for new or improperly renamed tests: - c.buildPassRateTestStats(testStats, float64(opts.PassRateRequiredNewTests)) - // If a new test reports no regression, and we're not using pass rate mode for all tests, we alter - // status to be missing basis for the pre-existing Fisher Exact behavior: - if testStats.ReportStatus == crtest.NotSignificant && opts.PassRateRequiredAllTests == 0 { - testStats.ReportStatus = crtest.MissingBasis - } - return - } else if opts.PassRateRequiredAllTests > 0 { - // If requested, switch to pass rate only testing to see what does not meet the criteria: - c.buildPassRateTestStats(testStats, float64(opts.PassRateRequiredAllTests)) - return - } - - // Otherwise we fall back to default behavior of Fishers Exact test: - c.buildFisherExactTestStats(testStats, logger) -} - -func (c *ComponentReportGenerator) buildFisherExactTestStats(testStats *testdetails.TestComparison, logger *log.Entry) { - - fisherExact := 0.0 - testStats.Comparison = crtest.FisherExact - - status := crtest.MissingBasis - opts := c.ReqOptions.AdvancedOption - if testStats.SampleStats.Total() == 0 { - if opts.IgnoreMissing { - status = crtest.NotSignificant - } else { - status = crtest.MissingSample - } - testStats.ReportStatus = status - testStats.FisherExact = thrift.Float64Ptr(0.0) - testStats.Explanations = append(testStats.Explanations, explanationNoRegression) - } else if testStats.BaseStats != nil && testStats.BaseStats.Total() != 0 { - samplePass := testStats.SampleStats.Passes(opts.FlakeAsFailure) - basePass := testStats.BaseStats.Passes(opts.FlakeAsFailure) - basisPassPercentage := float64(basePass) / float64(testStats.BaseStats.Total()) - effectivePityFactor := float64(opts.PityFactor) + testStats.PityAdjustment - effectiveMinimumFailure := opts.MinimumFailure - - // default starting status now that we know we have basis and sample - status = crtest.NotSignificant - - // now that we know sampleTotal is non zero - samplePassPercentage := float64(samplePass) / float64(testStats.SampleStats.Total()) - - // are we below the MinimumFailure threshold? - if effectiveMinimumFailure != 0 && - (testStats.SampleStats.Total()-samplePass) < effectiveMinimumFailure { - if status <= crtest.SignificantTriagedRegression { - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("%s regression detected.", crtest.StringForStatus(status))) - } - testStats.ReportStatus = status - testStats.FisherExact = thrift.Float64Ptr(0.0) - return - } - significant := false - improved := samplePassPercentage >= basisPassPercentage - - if improved { - // flip base and sample when improved - significant, fisherExact = c.fischerExactTest(testStats.RequiredConfidence, testStats.BaseStats.Total()-basePass, basePass, testStats.SampleStats.Total()-samplePass, samplePass) - } else if basisPassPercentage-samplePassPercentage > effectivePityFactor/100 { - significant, fisherExact = c.fischerExactTest(testStats.RequiredConfidence, testStats.SampleStats.Total()-samplePass, samplePass, testStats.BaseStats.Total()-basePass, basePass) - } - logger.Debugf("computed Fisher info: signifcant: %v, fisherExact: %v", significant, fisherExact) - if significant { - if improved { - status = crtest.SignificantImprovement - } else { - status = getRegressionStatus(basisPassPercentage, samplePassPercentage) - } - } - } - logger.Debugf("computed status: %d", int(status)) - testStats.ReportStatus = status - testStats.FisherExact = thrift.Float64Ptr(fisherExact) - - baseRelease := "no basis" - if testStats.BaseStats != nil { - baseRelease = testStats.BaseStats.Release - } - // If we have a regression, include explanations: - if testStats.ReportStatus <= crtest.SignificantTriagedRegression { - logger.Debugf("regression detected against: %s", baseRelease) - - if testStats.ReportStatus <= crtest.SignificantRegression { - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("%s regression detected.", crtest.StringForStatus(testStats.ReportStatus))) - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("Fishers Exact probability of a regression: %.2f%%.", float64(100)-*testStats.FisherExact)) - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("Test pass rate dropped from %.2f%% to %.2f%%.", - testStats.BaseStats.SuccessRate*float64(100), - testStats.SampleStats.SuccessRate*float64(100))) - } else { - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("%s regression detected.", crtest.StringForStatus(testStats.ReportStatus))) - } - } else { - logger.Debugf("NO regression detected against: %s", baseRelease) - } -} - -func (c *ComponentReportGenerator) buildPassRateTestStats(testStats *testdetails.TestComparison, requiredSuccessRate float64) { - - effectiveSuccessReq := requiredSuccessRate + testStats.RequiredPassRateAdjustment - - // Assume 2x our allowed failure rate = an extreme regression. - // i.e. if we require 90%, extreme is anything below 80% - // if we require 95%, extreme is anything below 90% - // if an adjustment is applied, still use the configured success rate to define extreme regression. - severeRegressionSuccessRate := effectiveSuccessReq - (100 - requiredSuccessRate) - - // Require 6 runs in the sample (typically 1 week for daily jobs) for us to consider a pass rate requirement for a new test: - sufficientRuns := testStats.SampleStats.Total() >= 6 - - opts := c.ReqOptions.AdvancedOption - successRate := testStats.SampleStats.PassRate(opts.FlakeAsFailure) - if sufficientRuns && successRate*100 < effectiveSuccessReq && testStats.SampleStats.FailureCount >= opts.MinimumFailure { - rStatus := crtest.SignificantRegression - if successRate*100 < severeRegressionSuccessRate { - rStatus = crtest.ExtremeRegression - } - testStats.ReportStatus = rStatus - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("Test has a %.2f%% pass rate, but %.2f%% is required.", successRate*100, effectiveSuccessReq)) - testStats.Comparison = crtest.PassRate - testStats.SampleStats.SuccessRate = successRate - return - } - - testStats.ReportStatus = crtest.NotSignificant - testStats.Explanations = append(testStats.Explanations, explanationNoRegression) -} - -func (c *ComponentReportGenerator) fischerExactTest(confidenceRequired, sampleFailure, sampleSuccess, baseFailure, baseSuccess int) (bool, float64) { - _, _, r, _ := fischer.FisherExactTest(sampleFailure, - sampleSuccess, - baseFailure, - baseSuccess) - return r < 1-float64(confidenceRequired)/100, r -} - func (c *ComponentReportGenerator) getUniqueJUnitColumnValuesLast60Days(ctx context.Context, field string, nested bool) ([]string, error) { diff --git a/pkg/api/componentreadiness/component_report_test.go b/pkg/api/componentreadiness/component_report_test.go index da05888b2..75ed1daa4 100644 --- a/pkg/api/componentreadiness/component_report_test.go +++ b/pkg/api/componentreadiness/component_report_test.go @@ -13,7 +13,6 @@ import ( "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" - "github.com/sirupsen/logrus" "github.com/stretchr/testify/assert" "github.com/openshift/sippy/pkg/api/componentreadiness/utils" @@ -1196,6 +1195,7 @@ func TestGenerateComponentReport(t *testing.T) { componentAndCapabilityGetter = fakeComponentAndCapabilityGetter for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { + tc.generator.initializeMiddleware() report, err := tc.generator.generateComponentTestReport(tc.baseStatus, tc.sampleStatus) assert.NoError(t, err, "error generating component report") @@ -1622,6 +1622,7 @@ func TestGenerateComponentTestDetailsReport(t *testing.T) { } t.Run(tc.name, func(t *testing.T) { + tc.generator.initializeMiddleware() report := tc.generator.internalGenerateTestDetailsReport("", nil, nil, baseStats, sampleStats, tc.generator.ReqOptions.TestIDOptions[0]) assert.Equal(t, tc.expectedReport.RowIdentification, report.RowIdentification, "expected report row identification %+v, got %+v", tc.expectedReport.RowIdentification, report.RowIdentification) assert.Equal(t, tc.expectedReport.ColumnIdentification, report.ColumnIdentification, "expected report column identification %+v, got %+v", tc.expectedReport.ColumnIdentification, report.ColumnIdentification) @@ -1682,7 +1683,7 @@ func Test_componentReportGenerator_normalizeProwJobName(t *testing.T) { } } -func Test_componentReportGenerator_assessComponentStatus(t *testing.T) { +func Test_componentReportGenerator_analyze(t *testing.T) { tests := []struct { name string sampleTotal int @@ -1824,6 +1825,7 @@ func Test_componentReportGenerator_assessComponentStatus(t *testing.T) { c.ReqOptions.AdvancedOption.PassRateRequiredNewTests = tt.requiredPassRateForNewTests c.ReqOptions.AdvancedOption.PassRateRequiredAllTests = tt.requiredPassRateForAllTests c.ReqOptions.AdvancedOption.MinimumFailure = tt.minFail + c.initializeMiddleware() testAnalysis := &testdetails.TestComparison{ SampleStats: testdetails.ReleaseStats{ @@ -1842,14 +1844,15 @@ func Test_componentReportGenerator_assessComponentStatus(t *testing.T) { }, } - c.assessComponentStatus(testAnalysis, logrus.NewEntry(logrus.New())) - assert.Equalf(t, tt.expectedStatus, testAnalysis.ReportStatus, "assessComponentStatus expected status not equal") + testKey := crtest.Identification{} + _, err := c.middlewares.Analyze(testKey, testAnalysis) + assert.NoError(t, err) + assert.Equalf(t, tt.expectedStatus, testAnalysis.ReportStatus, "Analyze expected status not equal") if tt.expectedFischers != nil { - // Mac and Linux do not matchup on floating point precision, so lets approximate the comparison: assert.Equalf(t, fmt.Sprintf("%.4f", *tt.expectedFischers), fmt.Sprintf("%.4f", *testAnalysis.FisherExact), - "assessComponentStatus expected fischers value not equal") + "Analyze expected fischers value not equal") } else { assert.Nil(t, testAnalysis.FisherExact) } diff --git a/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate.go b/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate.go new file mode 100644 index 000000000..10018bc8b --- /dev/null +++ b/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate.go @@ -0,0 +1,51 @@ +package alltestspassrate + +import ( + "context" + "sync" + + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/analysis" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +type AllTestsPassRate struct { + reqOptions reqopts.RequestOptions +} + +func NewAllTestsPassRateMiddleware(reqOptions reqopts.RequestOptions) *AllTestsPassRate { + return &AllTestsPassRate{reqOptions: reqOptions} +} + +func (a *AllTestsPassRate) Query(_ context.Context, _ *sync.WaitGroup, _ crtest.JobVariants, + _, _ chan map[string]crstatus.TestStatus, _ chan error) { +} + +func (a *AllTestsPassRate) QueryTestDetails(_ context.Context, _ *sync.WaitGroup, _ chan error, _ crtest.JobVariants) { +} + +func (a *AllTestsPassRate) PreAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (a *AllTestsPassRate) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (a *AllTestsPassRate) PreTestDetailsAnalysis(_ crtest.KeyWithVariants, _ *crstatus.TestJobRunStatuses) error { + return nil +} + +// Analyze claims all tests when PassRateRequiredAllTests is configured, applying a raw +// pass rate comparison instead of Fisher's exact test. +func (a *AllTestsPassRate) Analyze(_ crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { + opts := a.reqOptions.AdvancedOption + if opts.PassRateRequiredAllTests == 0 { + return false, nil + } + + analysis.BuildPassRateTestStats(testStats, float64(opts.PassRateRequiredAllTests), opts) + return true, nil +} diff --git a/pkg/api/componentreadiness/middleware/analysis/passrate.go b/pkg/api/componentreadiness/middleware/analysis/passrate.go new file mode 100644 index 000000000..764e508b6 --- /dev/null +++ b/pkg/api/componentreadiness/middleware/analysis/passrate.go @@ -0,0 +1,39 @@ +package analysis + +import ( + "fmt" + + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +const ExplanationNoRegression = "No significant regressions found" + +// BuildPassRateTestStats evaluates a test's sample pass rate against a required success rate +// and sets the report status accordingly. +func BuildPassRateTestStats(testStats *testdetails.TestComparison, requiredSuccessRate float64, opts reqopts.Advanced) { + effectiveSuccessReq := requiredSuccessRate + testStats.RequiredPassRateAdjustment + + // Assume 2x our allowed failure rate = an extreme regression. + severeRegressionSuccessRate := effectiveSuccessReq - (100 - requiredSuccessRate) + + sufficientRuns := testStats.SampleStats.Total() >= 6 + + successRate := testStats.SampleStats.PassRate(opts.FlakeAsFailure) + if sufficientRuns && successRate*100 < effectiveSuccessReq && testStats.SampleStats.FailureCount >= opts.MinimumFailure { + rStatus := crtest.SignificantRegression + if successRate*100 < severeRegressionSuccessRate { + rStatus = crtest.ExtremeRegression + } + testStats.ReportStatus = rStatus + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Test has a %.2f%% pass rate, but %.2f%% is required.", successRate*100, effectiveSuccessReq)) + testStats.Comparison = crtest.PassRate + testStats.SampleStats.SuccessRate = successRate + return + } + + testStats.ReportStatus = crtest.NotSignificant + testStats.Explanations = append(testStats.Explanations, ExplanationNoRegression) +} diff --git a/pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go b/pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go new file mode 100644 index 000000000..8240c374e --- /dev/null +++ b/pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go @@ -0,0 +1,153 @@ +package fishersexact + +import ( + "context" + "fmt" + "sync" + + "github.com/apache/thrift/lib/go/thrift" + fischer "github.com/glycerine/golang-fisher-exact" + log "github.com/sirupsen/logrus" + + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/analysis" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +type FishersExact struct { + reqOptions reqopts.RequestOptions +} + +func NewFishersExactMiddleware(reqOptions reqopts.RequestOptions) *FishersExact { + return &FishersExact{reqOptions: reqOptions} +} + +func (f *FishersExact) Query(_ context.Context, _ *sync.WaitGroup, _ crtest.JobVariants, + _, _ chan map[string]crstatus.TestStatus, _ chan error) { +} + +func (f *FishersExact) QueryTestDetails(_ context.Context, _ *sync.WaitGroup, _ chan error, _ crtest.JobVariants) { +} + +func (f *FishersExact) PreAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (f *FishersExact) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (f *FishersExact) PreTestDetailsAnalysis(_ crtest.KeyWithVariants, _ *crstatus.TestJobRunStatuses) error { + return nil +} + +// Analyze is the catch-all analysis middleware. It always claims the test and performs +// Fisher's exact test to determine regression significance. +func (f *FishersExact) Analyze(_ crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { + logger := log.WithField("middleware", "FishersExact") + opts := f.reqOptions.AdvancedOption + + if testStats.RequiredConfidence == 0 { + testStats.RequiredConfidence = opts.Confidence + } + + fisherExactVal := 0.0 + testStats.Comparison = crtest.FisherExact + + status := crtest.MissingBasis + if testStats.SampleStats.Total() == 0 { + if opts.IgnoreMissing { + status = crtest.NotSignificant + } else { + status = crtest.MissingSample + } + testStats.ReportStatus = status + testStats.FisherExact = thrift.Float64Ptr(0.0) + testStats.Explanations = append(testStats.Explanations, analysis.ExplanationNoRegression) + return true, nil + } else if testStats.BaseStats != nil && testStats.BaseStats.Total() != 0 { + samplePass := testStats.SampleStats.Passes(opts.FlakeAsFailure) + basePass := testStats.BaseStats.Passes(opts.FlakeAsFailure) + basisPassPercentage := float64(basePass) / float64(testStats.BaseStats.Total()) + effectivePityFactor := float64(opts.PityFactor) + testStats.PityAdjustment + effectiveMinimumFailure := opts.MinimumFailure + + status = crtest.NotSignificant + + samplePassPercentage := float64(samplePass) / float64(testStats.SampleStats.Total()) + + if effectiveMinimumFailure != 0 && + (testStats.SampleStats.Total()-samplePass) < effectiveMinimumFailure { + if status <= crtest.SignificantTriagedRegression { + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("%s regression detected.", crtest.StringForStatus(status))) + } + testStats.ReportStatus = status + testStats.FisherExact = thrift.Float64Ptr(0.0) + return true, nil + } + significant := false + improved := samplePassPercentage >= basisPassPercentage + + if improved { + significant, fisherExactVal = fisherExactTest(testStats.RequiredConfidence, testStats.BaseStats.Total()-basePass, basePass, testStats.SampleStats.Total()-samplePass, samplePass) + } else if basisPassPercentage-samplePassPercentage > effectivePityFactor/100 { + significant, fisherExactVal = fisherExactTest(testStats.RequiredConfidence, testStats.SampleStats.Total()-samplePass, samplePass, testStats.BaseStats.Total()-basePass, basePass) + } + logger.Debugf("computed Fisher info: signifcant: %v, fisherExact: %v", significant, fisherExactVal) + if significant { + if improved { + status = crtest.SignificantImprovement + } else { + status = getRegressionStatus(basisPassPercentage, samplePassPercentage) + } + } + } + logger.Debugf("computed status: %d", int(status)) + testStats.ReportStatus = status + testStats.FisherExact = thrift.Float64Ptr(fisherExactVal) + + baseRelease := "no basis" + if testStats.BaseStats != nil { + baseRelease = testStats.BaseStats.Release + } + if testStats.ReportStatus <= crtest.SignificantTriagedRegression { + logger.Debugf("regression detected against: %s", baseRelease) + + if testStats.ReportStatus <= crtest.SignificantRegression { + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("%s regression detected.", crtest.StringForStatus(testStats.ReportStatus))) + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Fishers Exact probability of a regression: %.2f%%.", float64(100)-*testStats.FisherExact)) + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Test pass rate dropped from %.2f%% to %.2f%%.", + testStats.BaseStats.SuccessRate*float64(100), + testStats.SampleStats.SuccessRate*float64(100))) + } else { + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("%s regression detected.", crtest.StringForStatus(testStats.ReportStatus))) + } + } else { + logger.Debugf("NO regression detected against: %s", baseRelease) + } + + return true, nil +} + +func fisherExactTest(confidenceRequired, sampleFailure, sampleSuccess, baseFailure, baseSuccess int) (bool, float64) { + _, _, r, _ := fischer.FisherExactTest(sampleFailure, + sampleSuccess, + baseFailure, + baseSuccess) + return r < 1-float64(confidenceRequired)/100, r +} + +func getRegressionStatus(basisPassPercentage, samplePassPercentage float64) crtest.Status { + if (basisPassPercentage - samplePassPercentage) > 0.15 { + return crtest.ExtremeRegression + } + + return crtest.SignificantRegression +} diff --git a/pkg/api/componentreadiness/middleware/interface.go b/pkg/api/componentreadiness/middleware/interface.go index db846a7c8..103da54be 100644 --- a/pkg/api/componentreadiness/middleware/interface.go +++ b/pkg/api/componentreadiness/middleware/interface.go @@ -34,6 +34,12 @@ type Middleware interface { // This allows for cheap reloads with fresh triage data without having to do an expensive report recalculation. PostAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error + // Analyze gives middleware the opportunity to perform statistical analysis for a test, + // replacing the default Fisher's exact / pass-rate logic. Return true if this middleware + // handled the analysis (first responder wins, subsequent middleware will not be called). + // Return false to defer to the next middleware or the default analysis. + Analyze(testKey crtest.Identification, testStats *testdetails.TestComparison) (bool, error) + // PreTestDetailsAnalysis gives middleware the opportunity to adjust inputs to the report status // prior to analysis. PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, status *crstatus.TestJobRunStatuses) error diff --git a/pkg/api/componentreadiness/middleware/linkinjector/linkinjector.go b/pkg/api/componentreadiness/middleware/linkinjector/linkinjector.go index 84bb66d4c..d644789f3 100644 --- a/pkg/api/componentreadiness/middleware/linkinjector/linkinjector.go +++ b/pkg/api/componentreadiness/middleware/linkinjector/linkinjector.go @@ -45,6 +45,10 @@ func (l *LinkInjector) PreAnalysis(testKey crtest.Identification, testStats *tes return nil } +func (l *LinkInjector) Analyze(_ crtest.Identification, _ *testdetails.TestComparison) (bool, error) { + return false, nil +} + // PostAnalysis injects HATEOAS links into test analysis results func (l *LinkInjector) PostAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error { // Early return if status is above FixedRegression (i.e. regression has not yet rolled off) diff --git a/pkg/api/componentreadiness/middleware/list.go b/pkg/api/componentreadiness/middleware/list.go index 5205677e3..95ee5e0b9 100644 --- a/pkg/api/componentreadiness/middleware/list.go +++ b/pkg/api/componentreadiness/middleware/list.go @@ -34,6 +34,19 @@ func (l List) PreAnalysis(testKey crtest.Identification, testStats *testdetails. return nil } +func (l List) Analyze(testKey crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { + for _, mw := range l { + handled, err := mw.Analyze(testKey, testStats) + if err != nil { + return false, err + } + if handled { + return true, nil + } + } + return false, nil +} + func (l List) PostAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error { for _, mw := range l { if err := mw.PostAnalysis(testKey, testStats); err != nil { diff --git a/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate.go b/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate.go new file mode 100644 index 000000000..749fe8c66 --- /dev/null +++ b/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate.go @@ -0,0 +1,56 @@ +package newtestpassrate + +import ( + "context" + "sync" + + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/analysis" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +type NewTestPassRate struct { + reqOptions reqopts.RequestOptions +} + +func NewNewTestPassRateMiddleware(reqOptions reqopts.RequestOptions) *NewTestPassRate { + return &NewTestPassRate{reqOptions: reqOptions} +} + +func (n *NewTestPassRate) Query(_ context.Context, _ *sync.WaitGroup, _ crtest.JobVariants, + _, _ chan map[string]crstatus.TestStatus, _ chan error) { +} + +func (n *NewTestPassRate) QueryTestDetails(_ context.Context, _ *sync.WaitGroup, _ chan error, _ crtest.JobVariants) { +} + +func (n *NewTestPassRate) PreAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (n *NewTestPassRate) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { + return nil +} + +func (n *NewTestPassRate) PreTestDetailsAnalysis(_ crtest.KeyWithVariants, _ *crstatus.TestJobRunStatuses) error { + return nil +} + +// Analyze claims tests that have no base stats (new tests) when PassRateRequiredNewTests is configured. +func (n *NewTestPassRate) Analyze(_ crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { + opts := n.reqOptions.AdvancedOption + if opts.PassRateRequiredNewTests == 0 { + return false, nil + } + if testStats.BaseStats != nil && testStats.BaseStats.Total() > 0 { + return false, nil + } + + analysis.BuildPassRateTestStats(testStats, float64(opts.PassRateRequiredNewTests), opts) + if testStats.ReportStatus == crtest.NotSignificant && opts.PassRateRequiredAllTests == 0 { + testStats.ReportStatus = crtest.MissingBasis + } + return true, nil +} diff --git a/pkg/api/componentreadiness/middleware/regressionallowances/regressionallowances.go b/pkg/api/componentreadiness/middleware/regressionallowances/regressionallowances.go index 7e79cdb43..5e7cc69ce 100644 --- a/pkg/api/componentreadiness/middleware/regressionallowances/regressionallowances.go +++ b/pkg/api/componentreadiness/middleware/regressionallowances/regressionallowances.go @@ -61,6 +61,10 @@ func (r *RegressionAllowances) PreAnalysis(testKey crtest.Identification, testSt return nil } +func (r *RegressionAllowances) Analyze(_ crtest.Identification, _ *testdetails.TestComparison) (bool, error) { + return false, nil +} + func (r *RegressionAllowances) PostAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error { return nil } diff --git a/pkg/api/componentreadiness/middleware/regressiontracker/regressiontracker.go b/pkg/api/componentreadiness/middleware/regressiontracker/regressiontracker.go index 124336d25..c5fc661eb 100644 --- a/pkg/api/componentreadiness/middleware/regressiontracker/regressiontracker.go +++ b/pkg/api/componentreadiness/middleware/regressiontracker/regressiontracker.go @@ -104,6 +104,10 @@ func (r *RegressionTracker) PreAnalysis(testKey crtest.Identification, testStats return nil } +func (r *RegressionTracker) Analyze(_ crtest.Identification, _ *testdetails.TestComparison) (bool, error) { + return false, nil +} + // PostAnalysis adjusts triages and status code (and thus icons) based on the triaged state of open regressions. func (r *RegressionTracker) PostAnalysis(testKey crtest.Identification, testStats *testdetails.TestComparison) error { if testStats.ReportStatus > crtest.SignificantTriagedRegression { diff --git a/pkg/api/componentreadiness/middleware/releasefallback/releasefallback.go b/pkg/api/componentreadiness/middleware/releasefallback/releasefallback.go index 25dc1d83b..3e7ee3cf9 100644 --- a/pkg/api/componentreadiness/middleware/releasefallback/releasefallback.go +++ b/pkg/api/componentreadiness/middleware/releasefallback/releasefallback.go @@ -67,8 +67,8 @@ type ReleaseFallback struct { releaseConfigs []v1.Release } -func (r *ReleaseFallback) Analyze(testID string, variants map[string]string, report *testdetails.TestComparison) error { - return nil +func (r *ReleaseFallback) Analyze(_ crtest.Identification, _ *testdetails.TestComparison) (bool, error) { + return false, nil } func (r *ReleaseFallback) Query(ctx context.Context, wg *sync.WaitGroup, allJobVariants crtest.JobVariants, diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index f8ae2c147..7c4c87449 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -169,15 +169,19 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup } } -// PreAnalysis overrides the normal fisher-exact statistical comparison for spot-check -// tests. Instead it applies a simple heuristic: any successful run in the sample window -// means the job is healthy (NotSignificant), zero successes means ExtremeRegression. -// Marks analysis as complete so the standard pipeline skips further processing. -func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, - testStats *testdetails.TestComparison) error { +func (s *SpotCheckJobs) PreAnalysis(_ crtest.Identification, + _ *testdetails.TestComparison) error { + return nil +} + +// Analyze claims spot-check tests and applies a simple heuristic: any successful run in +// the sample window means the job is healthy (NotSignificant), zero successes means +// ExtremeRegression. Returns false for non-spot-check tests to defer to other analyzers. +func (s *SpotCheckJobs) Analyze(testKey crtest.Identification, + testStats *testdetails.TestComparison) (bool, error) { if !isSpotCheckTestID(testKey.TestID) { - return nil + return false, nil } sampleDays := int(s.reqOptions.SpotCheckSample.End.Sub(s.reqOptions.SpotCheckSample.Start).Hours() / 24) @@ -194,10 +198,9 @@ func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, } testStats.Comparison = crtest.SpotCheck - testStats.AnalysisComplete = true testStats.BaseStats = nil - return nil + return true, nil } func (s *SpotCheckJobs) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { diff --git a/pkg/api/componentreadiness/test_details.go b/pkg/api/componentreadiness/test_details.go index 1262e2fb0..18b5f9605 100644 --- a/pkg/api/componentreadiness/test_details.go +++ b/pkg/api/componentreadiness/test_details.go @@ -477,7 +477,9 @@ func (c *ComponentReportGenerator) internalGenerateTestDetailsReport( logrus.WithError(err).Error("Failure from middleware analysis") } - c.assessComponentStatus(&testStats, log) + if _, err := c.middlewares.Analyze(testKey, &testStats); err != nil { + logrus.WithError(err).Error("Failure from middleware Analyze") + } report.TestComparison = testStats result.Analyses = []testdetails.Analysis{report} diff --git a/pkg/apis/api/componentreport/testdetails/types.go b/pkg/apis/api/componentreport/testdetails/types.go index 37e89c07a..c900e3ddb 100644 --- a/pkg/apis/api/componentreport/testdetails/types.go +++ b/pkg/apis/api/componentreport/testdetails/types.go @@ -79,10 +79,6 @@ type TestComparison struct { // LastFailure is the last time the regressed test failed. LastFailure *time.Time `json:"last_failure"` - // AnalysisComplete indicates middleware has already determined the report status. - // When true, assessComponentStatus skips its analysis (Fisher's, pass-rate, etc.). - AnalysisComplete bool `json:"-"` - // Regression is populated with data on when we first detected this regression. If unset it implies // the regression tracker has not yet run to find it, or you're using report params/a view without regression tracking. Regression *models.TestRegression `json:"regression,omitempty"` From 00972d9c8fe46d946f4474a213133dead38d2c4f Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Tue, 26 May 2026 11:19:09 -0300 Subject: [PATCH 09/37] Less risque package naming --- .../componentreadiness/component_report.go | 4 ++-- .../fisherexact.go} | 22 +++++++++---------- 2 files changed, 13 insertions(+), 13 deletions(-) rename pkg/api/componentreadiness/middleware/{fishersexact/fishersexact.go => fisherexact/fisherexact.go} (85%) diff --git a/pkg/api/componentreadiness/component_report.go b/pkg/api/componentreadiness/component_report.go index d04329d68..2e4af16b9 100644 --- a/pkg/api/componentreadiness/component_report.go +++ b/pkg/api/componentreadiness/component_report.go @@ -12,7 +12,7 @@ import ( "time" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/alltestspassrate" - "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/fishersexact" + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/fisherexact" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/linkinjector" "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/newtestpassrate" regressionallowances2 "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/regressionallowances" @@ -296,7 +296,7 @@ func (c *ComponentReportGenerator) initializeMiddleware() { // Analysis middleware ordered by priority — first responder wins. c.middlewares = append(c.middlewares, newtestpassrate.NewNewTestPassRateMiddleware(c.ReqOptions)) c.middlewares = append(c.middlewares, alltestspassrate.NewAllTestsPassRateMiddleware(c.ReqOptions)) - c.middlewares = append(c.middlewares, fishersexact.NewFishersExactMiddleware(c.ReqOptions)) + c.middlewares = append(c.middlewares, fisherexact.NewFisherExactMiddleware(c.ReqOptions)) // Initialize LinkInjector middleware linkInjector := linkinjector.NewLinkInjectorMiddleware(c.ReqOptions, c.baseURL) diff --git a/pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact.go similarity index 85% rename from pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go rename to pkg/api/componentreadiness/middleware/fisherexact/fisherexact.go index 8240c374e..f000dd107 100644 --- a/pkg/api/componentreadiness/middleware/fishersexact/fishersexact.go +++ b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact.go @@ -1,4 +1,4 @@ -package fishersexact +package fisherexact import ( "context" @@ -16,37 +16,37 @@ import ( "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" ) -type FishersExact struct { +type FisherExact struct { reqOptions reqopts.RequestOptions } -func NewFishersExactMiddleware(reqOptions reqopts.RequestOptions) *FishersExact { - return &FishersExact{reqOptions: reqOptions} +func NewFisherExactMiddleware(reqOptions reqopts.RequestOptions) *FisherExact { + return &FisherExact{reqOptions: reqOptions} } -func (f *FishersExact) Query(_ context.Context, _ *sync.WaitGroup, _ crtest.JobVariants, +func (f *FisherExact) Query(_ context.Context, _ *sync.WaitGroup, _ crtest.JobVariants, _, _ chan map[string]crstatus.TestStatus, _ chan error) { } -func (f *FishersExact) QueryTestDetails(_ context.Context, _ *sync.WaitGroup, _ chan error, _ crtest.JobVariants) { +func (f *FisherExact) QueryTestDetails(_ context.Context, _ *sync.WaitGroup, _ chan error, _ crtest.JobVariants) { } -func (f *FishersExact) PreAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { +func (f *FisherExact) PreAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { return nil } -func (f *FishersExact) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { +func (f *FisherExact) PostAnalysis(_ crtest.Identification, _ *testdetails.TestComparison) error { return nil } -func (f *FishersExact) PreTestDetailsAnalysis(_ crtest.KeyWithVariants, _ *crstatus.TestJobRunStatuses) error { +func (f *FisherExact) PreTestDetailsAnalysis(_ crtest.KeyWithVariants, _ *crstatus.TestJobRunStatuses) error { return nil } // Analyze is the catch-all analysis middleware. It always claims the test and performs // Fisher's exact test to determine regression significance. -func (f *FishersExact) Analyze(_ crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { - logger := log.WithField("middleware", "FishersExact") +func (f *FisherExact) Analyze(_ crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { + logger := log.WithField("middleware", "FisherExact") opts := f.reqOptions.AdvancedOption if testStats.RequiredConfidence == 0 { From aef4c42f24493aa0384048d38966a5879b7d22e5 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Tue, 26 May 2026 13:21:23 -0300 Subject: [PATCH 10/37] Human comments --- pkg/api/componentreadiness/component_report.go | 4 +++- .../middleware/spotcheckjobs/spotcheckjobs.go | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/pkg/api/componentreadiness/component_report.go b/pkg/api/componentreadiness/component_report.go index 2e4af16b9..47dfeefc4 100644 --- a/pkg/api/componentreadiness/component_report.go +++ b/pkg/api/componentreadiness/component_report.go @@ -277,7 +277,9 @@ func (c *ComponentReportGenerator) getCache() cache.Cache { func (c *ComponentReportGenerator) initializeMiddleware() { c.middlewares = middleware.List{} - // Spot-check jobs middleware runs first so synthetic results are in place for other middleware. + // Initialize all our middleware applicable to this request. + + // middlewares that inject synthetic tests must run first so results are in place for other middleware. if c.ReqOptions.SpotCheckSample != nil { c.middlewares = append(c.middlewares, spotcheckjobs.NewSpotCheckJobsMiddleware(c.dataProvider, c.ReqOptions)) } diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 7c4c87449..90b6a0477 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -250,7 +250,8 @@ func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, } // resolveComponentCapability maps a spot-check group to its component/capability -// using hardcoded patterns based on job names. This will be replaced by +// using hardcoded patterns based on job names. +// TODO: This will be replaced by // SpotCheckComponent/SpotCheckCapability variants once the variant registry is updated. func (s *SpotCheckJobs) resolveComponentCapability(group dataprovider.SpotCheckGroup) (string, string) { for _, jobName := range group.JobNames { From 1d18b8a95b978396965e334f4e2c6d34618cf9a2 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Wed, 27 May 2026 08:46:33 -0300 Subject: [PATCH 11/37] Fix muddled spot check jobs within variant combos --- .../dataprovider/bigquery/provider.go | 28 +++- .../dataprovider/interface.go | 4 + .../dataprovider/postgres/provider.go | 4 +- .../middleware/spotcheckjobs/spotcheckjobs.go | 132 ++++++++---------- 4 files changed, 91 insertions(+), 77 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 51443f467..a0ee6ddba 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -382,6 +382,7 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // synthetic test results without needing individual test case data. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + jobNameSubstrings []string, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { columnGroupByVariants := reqOptions.VariantOption.ColumnGroupBy @@ -439,6 +440,16 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions }) } + jobNameFilters := "" + for i, sub := range jobNameSubstrings { + paramName := fmt.Sprintf("jobNameSub_%d", i) + jobNameFilters += fmt.Sprintf(" AND LOWER(jobs.prowjob_job_name) LIKE CONCAT('%%', @%s, '%%')", paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: strings.ToLower(sub), + }) + } + queryString := fmt.Sprintf(` SELECT %s @@ -453,8 +464,9 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions AND jv_JobTier.variant_value = 'rare' AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s + %s GROUP BY %s - `, selectVariants, p.client.Dataset, joinVariants, variantFilters, + `, selectVariants, p.client.Dataset, joinVariants, variantFilters, jobNameFilters, groupByVariants) params = append(params, @@ -515,6 +527,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, + jobNameSubstrings []string, start, end time.Time) ([]dataprovider.JobRunDetail, error) { joinVariants := fmt.Sprintf( @@ -539,6 +552,16 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO }) } + jobNameFilters := "" + for i, sub := range jobNameSubstrings { + paramName := fmt.Sprintf("jobNameSub_%d", i) + jobNameFilters += fmt.Sprintf(" AND LOWER(jobs.prowjob_job_name) LIKE CONCAT('%%', @%s, '%%')", paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: strings.ToLower(sub), + }) + } + queryString := fmt.Sprintf(` SELECT jobs.prowjob_job_name AS job_name, @@ -554,8 +577,9 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO AND jv_JobTier.variant_value = 'rare' AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s + %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, variantFilters) + `, p.client.Dataset, joinVariants, variantFilters, jobNameFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, diff --git a/pkg/api/componentreadiness/dataprovider/interface.go b/pkg/api/componentreadiness/dataprovider/interface.go index 7f368e2a4..73b152ef3 100644 --- a/pkg/api/componentreadiness/dataprovider/interface.go +++ b/pkg/api/componentreadiness/dataprovider/interface.go @@ -70,15 +70,19 @@ type JobQuerier interface { type SpotCheckQuerier interface { // QuerySpotCheckJobRuns returns aggregated pass/fail per spot-check group, // grouped by variant columns. Queries the jobs table, not junit. + // jobNameSubstrings filters to jobs whose name contains ALL of the given substrings. QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + jobNameSubstrings []string, start, end time.Time) ([]SpotCheckGroup, error) // QuerySpotCheckJobRunDetails returns individual job runs for a specific // spot-check group, used for test details drill-down. + // jobNameSubstrings filters to jobs whose name contains ALL of the given substrings. QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, + jobNameSubstrings []string, start, end time.Time) ([]JobRunDetail, error) } diff --git a/pkg/api/componentreadiness/dataprovider/postgres/provider.go b/pkg/api/componentreadiness/dataprovider/postgres/provider.go index 508b261ac..f3808580c 100644 --- a/pkg/api/componentreadiness/dataprovider/postgres/provider.go +++ b/pkg/api/componentreadiness/dataprovider/postgres/provider.go @@ -801,11 +801,11 @@ func (p *PostgresProvider) LookupJobVariants(ctx context.Context, jobName string // Postgres does not support spot-check queries; these are BigQuery-only. func (p *PostgresProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + _ crtest.JobVariants, _ []string, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { return nil, nil } func (p *PostgresProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _ map[string]string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + _ crtest.JobVariants, _ map[string]string, _ []string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { return nil, nil } diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 90b6a0477..8aadc0f92 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -66,58 +66,53 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, return } - wg.Add(1) - go func() { - defer wg.Done() - select { - case <-ctx.Done(): - s.log.Info("context canceled during spot check query") - return - default: - } - - groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, - s.reqOptions, allJobVariants, - s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) - if err != nil { - errCh <- fmt.Errorf("spot check query failed: %w", err) - return - } - - sampleStatus := map[string]crstatus.TestStatus{} - for _, group := range groups { - component, capability := s.resolveComponentCapability(group) - if component == "" { - continue + for _, m := range spotCheckMappings { + wg.Go(func() { + select { + case <-ctx.Done(): + s.log.Info("context canceled during spot check query") + return + default: } - testKey := crtest.KeyWithVariants{ - TestID: syntheticTestID(component, capability), - Variants: group.Variants, + groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, + s.reqOptions, allJobVariants, m.substrings, + s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + if err != nil { + errCh <- fmt.Errorf("spot check query failed for %s/%s: %w", m.component, m.capability, err) + return } - keyStr := testKey.KeyOrDie() - atLeastOnePass := group.SuccessfulRuns >= 1 - successCount := 0 - if atLeastOnePass { - successCount = 1 - } + sampleStatus := map[string]crstatus.TestStatus{} + for _, group := range groups { + testKey := crtest.KeyWithVariants{ + TestID: syntheticTestID(m.component, m.capability), + Variants: group.Variants, + } + keyStr := testKey.KeyOrDie() + + atLeastOnePass := group.SuccessfulRuns >= 1 + successCount := 0 + if atLeastOnePass { + successCount = 1 + } - sampleStatus[keyStr] = crstatus.TestStatus{ - TestName: syntheticTestName(component, capability), - Component: component, - Capabilities: []string{capability}, - Variants: variantMapToSlice(group.Variants), - Count: crtest.Count{ - TotalCount: 1, - SuccessCount: successCount, - }, + sampleStatus[keyStr] = crstatus.TestStatus{ + TestName: syntheticTestName(m.component, m.capability), + Component: m.component, + Capabilities: []string{m.capability}, + Variants: variantMapToSlice(group.Variants), + Count: crtest.Count{ + TotalCount: 1, + SuccessCount: successCount, + }, + } } - } - s.log.Infof("injecting %d spot-check synthetic test results", len(sampleStatus)) - sampleStatusCh <- sampleStatus - }() + s.log.Infof("injecting %d spot-check synthetic test results for %s/%s", len(sampleStatus), m.component, m.capability) + sampleStatusCh <- sampleStatus + }) + } } // QueryTestDetails fetches individual job run details for spot-check synthetic tests, @@ -134,10 +129,13 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup continue } - opt := testIDOpt - wg.Add(1) - go func() { - defer wg.Done() + m := mappingForTestID(testIDOpt.TestID) + if m == nil { + s.log.Warnf("no mapping found for spot-check test ID %s", testIDOpt.TestID) + continue + } + + wg.Go(func() { select { case <-ctx.Done(): return @@ -146,7 +144,7 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, s.reqOptions, allJobVariants, - opt.RequestedVariants, + testIDOpt.RequestedVariants, m.substrings, s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) if err != nil { errCh <- fmt.Errorf("spot check details query failed: %w", err) @@ -154,8 +152,8 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup } testKey := crtest.KeyWithVariants{ - TestID: opt.TestID, - Variants: opt.RequestedVariants, + TestID: testIDOpt.TestID, + Variants: testIDOpt.RequestedVariants, } s.sampleJobDetailsMutex.Lock() @@ -164,8 +162,8 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup s.sampleJobDetails = map[string][]dataprovider.JobRunDetail{} } s.sampleJobDetails[testKey.KeyOrDie()] = details - s.log.Infof("loaded %d spot-check job run details for %s", len(details), opt.TestID) - }() + s.log.Infof("loaded %d spot-check job run details for %s", len(details), testIDOpt.TestID) + }) } } @@ -249,27 +247,15 @@ func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, return nil } -// resolveComponentCapability maps a spot-check group to its component/capability -// using hardcoded patterns based on job names. -// TODO: This will be replaced by -// SpotCheckComponent/SpotCheckCapability variants once the variant registry is updated. -func (s *SpotCheckJobs) resolveComponentCapability(group dataprovider.SpotCheckGroup) (string, string) { - for _, jobName := range group.JobNames { - lower := strings.ToLower(jobName) - for _, m := range spotCheckMappings { - allMatch := true - for _, sub := range m.substrings { - if !strings.Contains(lower, sub) { - allMatch = false - break - } - } - if allMatch { - return m.component, m.capability - } +// mappingForTestID returns the spotCheckMapping that produced the given synthetic testID, +// or nil if no mapping matches. +func mappingForTestID(testID string) *spotCheckMapping { + for i, m := range spotCheckMappings { + if syntheticTestID(m.component, m.capability) == testID { + return &spotCheckMappings[i] } } - return "", "" + return nil } func isSpotCheckTestID(testID string) bool { From 4d9a4f2a3ea52b43fccdb1e4a132ecc441659e78 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Mon, 8 Jun 2026 08:24:14 -0300 Subject: [PATCH 12/37] Fix a merge compile error --- pkg/api/componentreadiness/utils/queryparamparser.go | 2 +- pkg/dataloader/regressioncacheloader/regressioncacheloader.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index 6f70c6968..60caa9662 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -62,7 +62,7 @@ func ParseComponentReportRequest( RelativeStart: view.SpotCheckSample.RelativeStart, RelativeEnd: view.SpotCheckSample.RelativeEnd, } - resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", spotCheckRelative, crTimeRoundingFactor) + resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", spotCheckRelative, crTimeRoundingFactor, crTimeRoundingOffset) if resolveErr != nil { err = resolveErr return diff --git a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go index ce93a9c2c..3dd141470 100644 --- a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go +++ b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go @@ -512,7 +512,7 @@ func (l *RegressionCacheLoader) buildGenerator( RelativeEnd: view.SpotCheckSample.RelativeEnd, } resolved, err := utils.GetViewReleaseOptions( - l.releases, "spot_check", spotCheckRelative, cacheOpts.CRTimeRoundingFactor) + l.releases, "spot_check", spotCheckRelative, cacheOpts.CRTimeRoundingFactor, cacheOpts.CRTimeRoundingOffset) if err != nil { return nil, err } From 2770e88713dc52873abdc4ac9071a8de59c23ddb Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Mon, 8 Jun 2026 11:27:56 -0300 Subject: [PATCH 13/37] Refactor how variants opt a job into spot check status --- config/views.yaml | 1 + .../dataprovider/bigquery/provider.go | 101 +++--- .../dataprovider/interface.go | 14 +- .../dataprovider/postgres/provider.go | 4 +- .../middleware/spotcheckjobs/spotcheckjobs.go | 145 ++++----- pkg/variantregistry/ocp.go | 116 ++++--- pkg/variantregistry/ocp_test.go | 66 +++- pkg/variantregistry/snapshot.yaml | 288 +++++++++++++----- 8 files changed, 503 insertions(+), 232 deletions(-) diff --git a/config/views.yaml b/config/views.yaml index 8ba683cd1..d5280becf 100755 --- a/config/views.yaml +++ b/config/views.yaml @@ -43,6 +43,7 @@ component_readiness: - blocking - informing - standard + - spotcheck LayeredProduct: - none - virt diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index a0ee6ddba..e52cbdd0f 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -376,13 +376,30 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // --- SpotCheckQuerier --- -// QuerySpotCheckJobRuns queries the jobs table (not junit) for rare-tier periodic/release -// jobs, grouping by the requested variant columns (e.g. Platform, Architecture, Network). -// Returns aggregated pass/fail counts per variant group so the middleware can create +// spotCheckFallbackSQL provides a CASE expression for deriving component/capability +// from job names when the SpotCheckComponent/SpotCheckCapability variants are not yet +// populated in the job_variants table. This fallback can be removed once the variant +// syncer has run and all spot-check jobs have the new variants. +const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'etcd' + END)` + +const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'CPU Partitioning' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Scaling' + END)` + +// QuerySpotCheckJobRuns queries the jobs table (not junit) for spotcheck-tier periodic/release +// jobs, grouping by component, capability, and the requested variant columns. +// Returns aggregated pass/fail counts per group so the middleware can create // synthetic test results without needing individual test case data. +// During the transition period, also matches jobs with the legacy 'rare' tier and uses +// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, - jobNameSubstrings []string, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { columnGroupByVariants := reqOptions.VariantOption.ColumnGroupBy @@ -402,19 +419,25 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions } groupByVariants := strings.Join(groupByParts, ", ") - // Always join Release and JobTier + // Always join Release, JobTier, SpotCheckComponent, SpotCheckCapability joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_Release ON jobs.prowjob_job_name = jv_Release.job_name AND jv_Release.variant_name = 'Release'\n", p.client.Dataset) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_SpotCheckComponent ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent'\n", + p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_SpotCheckCapability ON jobs.prowjob_job_name = jv_SpotCheckCapability.job_name AND jv_SpotCheckCapability.variant_name = 'SpotCheckCapability'\n", + p.client.Dataset) // Track which variant groups already have JOINs joinedVariants := sets.NewString(columnGroupByVariants.List()...) - joinedVariants.Insert("Release", "JobTier") + joinedVariants.Insert("Release", "JobTier", "SpotCheckComponent", "SpotCheckCapability") - // Build include variant filters (Platform, Architecture, etc.) but skip JobTier + // Build include variant filters (Platform, Architecture, etc.) but skip JobTier and SpotCheck variants variantFilters := "" var params []bigquery.QueryParameter includeVariants := reqOptions.VariantOption.IncludeVariants @@ -422,7 +445,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions includeVariants = map[string][]string{} } for _, group := range sortedKeys(includeVariants) { - if group == "JobTier" { + if group == "JobTier" || group == "SpotCheckComponent" || group == "SpotCheckCapability" { continue } cleanGroup := param.Cleanse(group) @@ -440,19 +463,11 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions }) } - jobNameFilters := "" - for i, sub := range jobNameSubstrings { - paramName := fmt.Sprintf("jobNameSub_%d", i) - jobNameFilters += fmt.Sprintf(" AND LOWER(jobs.prowjob_job_name) LIKE CONCAT('%%', @%s, '%%')", paramName) - params = append(params, bigquery.QueryParameter{ - Name: paramName, - Value: strings.ToLower(sub), - }) - } - queryString := fmt.Sprintf(` SELECT %s + %s AS spot_check_component, + %s AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names @@ -461,12 +476,13 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value = 'rare' + AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s - %s - GROUP BY %s - `, selectVariants, p.client.Dataset, joinVariants, variantFilters, jobNameFilters, + GROUP BY %s, spot_check_component, spot_check_capability + HAVING spot_check_component IS NOT NULL AND spot_check_capability IS NOT NULL + `, selectVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, + p.client.Dataset, joinVariants, variantFilters, groupByVariants) params = append(params, @@ -497,6 +513,12 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions group := dataprovider.SpotCheckGroup{ Variants: map[string]string{}, } + if val, ok := rawRow["spot_check_component"]; ok && val != nil { + group.Component = val.(string) + } + if val, ok := rawRow["spot_check_capability"]; ok && val != nil { + group.Capability = val.(string) + } for _, v := range columnGroupByVariants.List() { cleanV := param.Cleanse(v) if val, ok := rawRow["variant_"+cleanV]; ok && val != nil { @@ -522,12 +544,14 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions } // QuerySpotCheckJobRunDetails returns individual job runs matching the given variant -// filters, used to populate the test details drill-down page for a specific spot-check -// synthetic test. +// and component/capability filters, used to populate the test details drill-down page +// for a specific spot-check synthetic test. +// During the transition period, also matches jobs with the legacy 'rare' tier and uses +// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, - jobNameSubstrings []string, + component, capability string, start, end time.Time) ([]dataprovider.JobRunDetail, error) { joinVariants := fmt.Sprintf( @@ -536,6 +560,12 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_SpotCheckComponent ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent'\n", + p.client.Dataset) + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_SpotCheckCapability ON jobs.prowjob_job_name = jv_SpotCheckCapability.job_name AND jv_SpotCheckCapability.variant_name = 'SpotCheckCapability'\n", + p.client.Dataset) variantFilters := "" var params []bigquery.QueryParameter @@ -552,16 +582,6 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO }) } - jobNameFilters := "" - for i, sub := range jobNameSubstrings { - paramName := fmt.Sprintf("jobNameSub_%d", i) - jobNameFilters += fmt.Sprintf(" AND LOWER(jobs.prowjob_job_name) LIKE CONCAT('%%', @%s, '%%')", paramName) - params = append(params, bigquery.QueryParameter{ - Name: paramName, - Value: strings.ToLower(sub), - }) - } - queryString := fmt.Sprintf(` SELECT jobs.prowjob_job_name AS job_name, @@ -574,17 +594,22 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value = 'rare' + AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') - %s + AND LOWER(%s) = LOWER(@SpotCheckComponent) + AND LOWER(%s) = LOWER(@SpotCheckCapability) %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, variantFilters, jobNameFilters) + `, p.client.Dataset, joinVariants, + spotCheckComponentFallback, spotCheckCapabilityFallback, + variantFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, bigquery.QueryParameter{Name: "To", Value: end}, bigquery.QueryParameter{Name: "Release", Value: reqOptions.SampleRelease.Name}, + bigquery.QueryParameter{Name: "SpotCheckComponent", Value: component}, + bigquery.QueryParameter{Name: "SpotCheckCapability", Value: capability}, ) q := p.client.Query(ctx, bqlabel.CRSpotCheckDetails, queryString) diff --git a/pkg/api/componentreadiness/dataprovider/interface.go b/pkg/api/componentreadiness/dataprovider/interface.go index 73b152ef3..af0def7d6 100644 --- a/pkg/api/componentreadiness/dataprovider/interface.go +++ b/pkg/api/componentreadiness/dataprovider/interface.go @@ -69,20 +69,20 @@ type JobQuerier interface { // SpotCheckQuerier fetches job-level pass/fail data for spot-check analysis. type SpotCheckQuerier interface { // QuerySpotCheckJobRuns returns aggregated pass/fail per spot-check group, - // grouped by variant columns. Queries the jobs table, not junit. - // jobNameSubstrings filters to jobs whose name contains ALL of the given substrings. + // grouped by SpotCheckComponent, SpotCheckCapability, and the column group-by variants. + // Queries the jobs table, not junit. During the transition period, falls back to + // job name substring matching when SpotCheckComponent/SpotCheckCapability variants + // are not yet populated in the job_variants table. QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, - jobNameSubstrings []string, start, end time.Time) ([]SpotCheckGroup, error) // QuerySpotCheckJobRunDetails returns individual job runs for a specific // spot-check group, used for test details drill-down. - // jobNameSubstrings filters to jobs whose name contains ALL of the given substrings. QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, - jobNameSubstrings []string, + component, capability string, start, end time.Time) ([]JobRunDetail, error) } @@ -108,8 +108,10 @@ type JobRunStats struct { } // SpotCheckGroup contains aggregated pass/fail for a set of spot-check jobs -// sharing the same variant column values. +// sharing the same component, capability, and variant column values. type SpotCheckGroup struct { + Component string `json:"component"` + Capability string `json:"capability"` Variants map[string]string `json:"variants"` TotalRuns int `json:"total_runs"` SuccessfulRuns int `json:"successful_runs"` diff --git a/pkg/api/componentreadiness/dataprovider/postgres/provider.go b/pkg/api/componentreadiness/dataprovider/postgres/provider.go index f3808580c..a7bde3cbb 100644 --- a/pkg/api/componentreadiness/dataprovider/postgres/provider.go +++ b/pkg/api/componentreadiness/dataprovider/postgres/provider.go @@ -801,11 +801,11 @@ func (p *PostgresProvider) LookupJobVariants(ctx context.Context, jobName string // Postgres does not support spot-check queries; these are BigQuery-only. func (p *PostgresProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _ []string, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { return nil, nil } func (p *PostgresProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _ map[string]string, _ []string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + _ crtest.JobVariants, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { return nil, nil } diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 8aadc0f92..8f18a328c 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -17,23 +17,14 @@ import ( var _ middleware.Middleware = &SpotCheckJobs{} -// spotCheckMapping defines the hardcoded component/capability for spot-check jobs -// until SpotCheckComponent/SpotCheckCapability variants exist in the variant registry. -type spotCheckMapping struct { - substrings []string - component string - capability string -} - -var spotCheckMappings = []spotCheckMapping{ - {substrings: []string{"-cpu-partitioning"}, component: "Node", capability: "CPU Partitioning"}, - {substrings: []string{"-etcd-scaling"}, component: "etcd", capability: "Scaling"}, -} - // NewSpotCheckJobsMiddleware creates middleware that injects synthetic test results for -// "rare" tier jobs (e.g. cpu-partitioning, etcd-scaling) that don't run in the standard +// spot-check jobs (e.g. cpu-partitioning, etcd-scaling) that don't run in the standard // junit-based test pipeline. These jobs are evaluated on a simple pass/fail basis: // at least one successful run in the sample window means healthy. +// +// Jobs are identified by the SpotCheckComponent and SpotCheckCapability variants in the +// variant registry. The component/capability values control where the synthetic test +// results appear in the component readiness report. func NewSpotCheckJobsMiddleware( provider dataprovider.DataProvider, reqOptions reqopts.RequestOptions, @@ -56,8 +47,9 @@ type SpotCheckJobs struct { } // Query fetches aggregated spot-check job results from BigQuery, creates synthetic -// test statuses (one per variant group), and injects them into the sample status channel. -// Each synthetic test uses a binary pass/fail: >=1 successful run = pass. +// test statuses (one per component/capability/variant group), and injects them into +// the sample status channel. Each synthetic test uses a binary pass/fail: +// >=1 successful run = pass. func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, allJobVariants crtest.JobVariants, _, sampleStatusCh chan map[string]crstatus.TestStatus, errCh chan error) { @@ -66,53 +58,56 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, return } - for _, m := range spotCheckMappings { - wg.Go(func() { - select { - case <-ctx.Done(): - s.log.Info("context canceled during spot check query") - return - default: + wg.Go(func() { + select { + case <-ctx.Done(): + s.log.Info("context canceled during spot check query") + return + default: + } + + groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, + s.reqOptions, allJobVariants, + s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + if err != nil { + errCh <- fmt.Errorf("spot check query failed: %w", err) + return + } + + sampleStatus := map[string]crstatus.TestStatus{} + for _, group := range groups { + if group.Component == "" || group.Capability == "" { + s.log.Warnf("skipping spot-check group with empty component/capability: %+v", group) + continue } - groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, - s.reqOptions, allJobVariants, m.substrings, - s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) - if err != nil { - errCh <- fmt.Errorf("spot check query failed for %s/%s: %w", m.component, m.capability, err) - return + testKey := crtest.KeyWithVariants{ + TestID: syntheticTestID(group.Component, group.Capability), + Variants: group.Variants, } + keyStr := testKey.KeyOrDie() - sampleStatus := map[string]crstatus.TestStatus{} - for _, group := range groups { - testKey := crtest.KeyWithVariants{ - TestID: syntheticTestID(m.component, m.capability), - Variants: group.Variants, - } - keyStr := testKey.KeyOrDie() - - atLeastOnePass := group.SuccessfulRuns >= 1 - successCount := 0 - if atLeastOnePass { - successCount = 1 - } - - sampleStatus[keyStr] = crstatus.TestStatus{ - TestName: syntheticTestName(m.component, m.capability), - Component: m.component, - Capabilities: []string{m.capability}, - Variants: variantMapToSlice(group.Variants), - Count: crtest.Count{ - TotalCount: 1, - SuccessCount: successCount, - }, - } + atLeastOnePass := group.SuccessfulRuns >= 1 + successCount := 0 + if atLeastOnePass { + successCount = 1 } - s.log.Infof("injecting %d spot-check synthetic test results for %s/%s", len(sampleStatus), m.component, m.capability) - sampleStatusCh <- sampleStatus - }) - } + sampleStatus[keyStr] = crstatus.TestStatus{ + TestName: syntheticTestName(group.Component, group.Capability), + Component: group.Component, + Capabilities: []string{group.Capability}, + Variants: variantMapToSlice(group.Variants), + Count: crtest.Count{ + TotalCount: 1, + SuccessCount: successCount, + }, + } + } + + s.log.Infof("injecting %d spot-check synthetic test results", len(sampleStatus)) + sampleStatusCh <- sampleStatus + }) } // QueryTestDetails fetches individual job run details for spot-check synthetic tests, @@ -129,9 +124,9 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup continue } - m := mappingForTestID(testIDOpt.TestID) - if m == nil { - s.log.Warnf("no mapping found for spot-check test ID %s", testIDOpt.TestID) + component, capability := componentCapabilityFromTestID(testIDOpt.TestID) + if component == "" || capability == "" { + s.log.Warnf("could not parse component/capability from spot-check test ID %s", testIDOpt.TestID) continue } @@ -144,7 +139,7 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, s.reqOptions, allJobVariants, - testIDOpt.RequestedVariants, m.substrings, + testIDOpt.RequestedVariants, component, capability, s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) if err != nil { errCh <- fmt.Errorf("spot check details query failed: %w", err) @@ -247,21 +242,27 @@ func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, return nil } -// mappingForTestID returns the spotCheckMapping that produced the given synthetic testID, -// or nil if no mapping matches. -func mappingForTestID(testID string) *spotCheckMapping { - for i, m := range spotCheckMappings { - if syntheticTestID(m.component, m.capability) == testID { - return &spotCheckMappings[i] - } - } - return nil -} - func isSpotCheckTestID(testID string) bool { return strings.HasPrefix(testID, "spotcheck:") } +// componentCapabilityFromTestID extracts the component and capability from a synthetic +// spot-check test ID. The format is "spotcheck::" where the +// component is lowercased and capability has spaces replaced with dashes. +func componentCapabilityFromTestID(testID string) (string, string) { + parts := strings.SplitN(testID, ":", 3) + if len(parts) != 3 { + return "", "" + } + // The test ID stores lowercased component and dash-separated capability. + // We need to convert back to the original format for the BigQuery query. + // The COALESCE fallback in the query matches exact component/capability values, + // so we need title case for component and space-separated for capability. + component := parts[1] + capability := strings.ReplaceAll(parts[2], "-", " ") + return component, capability +} + func syntheticTestID(component, capability string) string { return fmt.Sprintf("spotcheck:%s:%s", strings.ToLower(component), diff --git a/pkg/variantregistry/ocp.go b/pkg/variantregistry/ocp.go index e542b48a3..ddc786051 100644 --- a/pkg/variantregistry/ocp.go +++ b/pkg/variantregistry/ocp.go @@ -427,34 +427,36 @@ var ( ) const ( - VariantAggregation = "Aggregation" // aggregated or none - VariantArch = "Architecture" - VariantFeatureSet = "FeatureSet" // techpreview / standard - VariantInstaller = "Installer" // ipi / upi / assisted - VariantNetwork = "Network" - VariantNetworkAccess = "NetworkAccess" // disconnected / proxy / standard - VariantNetworkStack = "NetworkStack" // ipv4 / ipv6 / dual - VariantOwner = "Owner" // eng / osd - VariantPlatform = "Platform" - VariantScheduler = "Scheduler" // realtime / standard - VariantSecurityMode = "SecurityMode" // fips / default - VariantSuite = "Suite" // parallel / serial - VariantProcedure = "Procedure" // for jobs that do a specific procedure on the cluster (etcd scaling, cpu partitioning, etc.), and then optionally run conformance - VariantJobTier = "JobTier" // specifies rare, blocking, informing, standard jobs - VariantTopology = "Topology" // ha / single / compact / external - VariantUpgrade = "Upgrade" - VariantContainerRuntime = "ContainerRuntime" // runc / crun - VariantCGroupMode = "CGroupMode" // v2 / v1 - VariantRelease = "Release" - VariantReleaseMinor = "ReleaseMinor" - VariantReleaseMajor = "ReleaseMajor" - VariantFromRelease = "FromRelease" - VariantFromReleaseMinor = "FromReleaseMinor" - VariantFromReleaseMajor = "FromReleaseMajor" - VariantLayeredProduct = "LayeredProduct" - VariantOS = "OS" - VariantDefaultValue = "default" - VariantNoValue = "none" + VariantAggregation = "Aggregation" // aggregated or none + VariantArch = "Architecture" + VariantFeatureSet = "FeatureSet" // techpreview / standard + VariantInstaller = "Installer" // ipi / upi / assisted + VariantNetwork = "Network" + VariantNetworkAccess = "NetworkAccess" // disconnected / proxy / standard + VariantNetworkStack = "NetworkStack" // ipv4 / ipv6 / dual + VariantOwner = "Owner" // eng / osd + VariantPlatform = "Platform" + VariantScheduler = "Scheduler" // realtime / standard + VariantSecurityMode = "SecurityMode" // fips / default + VariantSuite = "Suite" // parallel / serial + VariantProcedure = "Procedure" // for jobs that do a specific procedure on the cluster (etcd scaling, cpu partitioning, etc.), and then optionally run conformance + VariantJobTier = "JobTier" // specifies rare, blocking, informing, standard jobs + VariantTopology = "Topology" // ha / single / compact / external + VariantUpgrade = "Upgrade" + VariantContainerRuntime = "ContainerRuntime" // runc / crun + VariantCGroupMode = "CGroupMode" // v2 / v1 + VariantRelease = "Release" + VariantReleaseMinor = "ReleaseMinor" + VariantReleaseMajor = "ReleaseMajor" + VariantFromRelease = "FromRelease" + VariantFromReleaseMinor = "FromReleaseMinor" + VariantFromReleaseMajor = "FromReleaseMajor" + VariantLayeredProduct = "LayeredProduct" + VariantOS = "OS" + VariantSpotCheckComponent = "SpotCheckComponent" // component readiness component for spot-check jobs + VariantSpotCheckCapability = "SpotCheckCapability" // component readiness capability for spot-check jobs + VariantDefaultValue = "default" + VariantNoValue = "none" ) func (v *OCPVariantLoader) IdentifyVariants(jLog logrus.FieldLogger, jobName string) map[string]string { @@ -480,6 +482,7 @@ func (v *OCPVariantLoader) IdentifyVariants(jLog logrus.FieldLogger, jobName str setContainerRuntime, setProcedure, setOS, + setSpotCheckClassification, v.setJobTier, // Keep this near last, it relies on other variants like owner } { setter(jLog, variants, jobName) @@ -726,29 +729,64 @@ func (v *OCPVariantLoader) setRelease(logger logrus.FieldLogger, variants map[st } } +// setSpotCheckClassification identifies jobs that should be evaluated as spot-check jobs +// in Component Readiness. These jobs run infrequently ("rare" tier historically) and are +// assessed on a simple pass/fail basis rather than per-test-case junit analysis. +// The SpotCheckComponent and SpotCheckCapability variants control where these synthetic +// results appear in the component readiness report. +func setSpotCheckClassification(_ logrus.FieldLogger, variants map[string]string, jobName string) { + jobNameLower := strings.ToLower(jobName) + + spotCheckPatterns := []struct { + substrings []string + component string + capability string + }{ + {[]string{"-cpu-partitioning"}, "Node", "CPU Partitioning"}, + {[]string{"-etcd-scaling"}, "etcd", "Scaling"}, + } + + for _, p := range spotCheckPatterns { + allMatch := true + for _, sub := range p.substrings { + if !strings.Contains(jobNameLower, sub) { + allMatch = false + break + } + } + if allMatch { + variants[VariantSpotCheckComponent] = p.component + variants[VariantSpotCheckCapability] = p.capability + return + } + } +} + // setJobTier sets the jobTier for a job, with values like this: // -// blocking: blocking job on payloads, covered by component readiness -// informing: informing job on payloads, covered by component readiness -// standard: should be visible in default views (component readiness, sippy), covered by component readiness -// rare: highly reliable jobs that run at a reduced frequency -// candidate: not covered by component readiness, but may be promoted in the future -// hidden: data should still be synced, but not shown by default -// excluded: data should not be synced, and excluded from all views +// blocking: blocking job on payloads, covered by component readiness +// informing: informing job on payloads, covered by component readiness +// standard: should be visible in default views (component readiness, sippy), covered by component readiness +// spotcheck: jobs evaluated by spot-check analysis (job pass/fail, not junit); views opt in via JobTier include +// candidate: not covered by component readiness, but may be promoted in the future +// hidden: data should still be synced, but not shown by default +// excluded: data should not be synced, and excluded from all views // // Note: blocking/informing/standard tiers may be downgraded to candidate by // adjustJobTierBasedOnView if the job's variants don't match the release-main view. func (v *OCPVariantLoader) setJobTier(_ logrus.FieldLogger, variants map[string]string, jobName string) { + // Jobs classified as spot-check get the spotcheck tier automatically. + if _, ok := variants[VariantSpotCheckComponent]; ok { + variants[VariantJobTier] = "spotcheck" + return + } + jobNameLower := strings.ToLower(jobName) jobTierPatterns := []struct { substrings []string jobTier string }{ - // Rarely run - {[]string{"-cpu-partitioning"}, "rare"}, - {[]string{"-etcd-scaling"}, "rare"}, - // QE jobs allowlisted for Component Readiness {[]string{"-automated-release"}, "standard"}, diff --git a/pkg/variantregistry/ocp_test.go b/pkg/variantregistry/ocp_test.go index 338d5ab22..219b2701d 100644 --- a/pkg/variantregistry/ocp_test.go +++ b/pkg/variantregistry/ocp_test.go @@ -2176,6 +2176,66 @@ func TestVariantSyncer(t *testing.T) { VariantOS: "unknown", }, }, + { + job: "periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-cpu-partitioning", + expected: map[string]string{ + VariantRelease: "4.18", + VariantReleaseMajor: "4", + VariantReleaseMinor: "18", + VariantArch: "amd64", + VariantInstaller: "ipi", + VariantPlatform: "aws", + VariantNetwork: "ovn", + VariantNetworkStack: "ipv4", + VariantOwner: "eng", + VariantTopology: "ha", + VariantSuite: "unknown", + VariantUpgrade: VariantNoValue, + VariantProcedure: "cpu-partitioning", + VariantJobTier: "spotcheck", + VariantAggregation: VariantNoValue, + VariantSecurityMode: VariantDefaultValue, + VariantFeatureSet: VariantDefaultValue, + VariantNetworkAccess: VariantDefaultValue, + VariantScheduler: VariantDefaultValue, + VariantContainerRuntime: "crun", + VariantCGroupMode: "v2", + VariantLayeredProduct: VariantNoValue, + VariantOS: "rhcos9", + VariantSpotCheckComponent: "Node", + VariantSpotCheckCapability: "CPU Partitioning", + }, + }, + { + job: "periodic-ci-openshift-release-main-nightly-4.18-e2e-gcp-ovn-etcd-scaling", + expected: map[string]string{ + VariantRelease: "4.18", + VariantReleaseMajor: "4", + VariantReleaseMinor: "18", + VariantArch: "amd64", + VariantInstaller: "ipi", + VariantPlatform: "gcp", + VariantNetwork: "ovn", + VariantNetworkStack: "ipv4", + VariantOwner: "eng", + VariantTopology: "ha", + VariantSuite: "etcd-scaling", + VariantUpgrade: VariantNoValue, + VariantProcedure: "etcd-scaling", + VariantJobTier: "spotcheck", + VariantAggregation: VariantNoValue, + VariantSecurityMode: VariantDefaultValue, + VariantFeatureSet: VariantDefaultValue, + VariantNetworkAccess: VariantDefaultValue, + VariantScheduler: VariantDefaultValue, + VariantContainerRuntime: "crun", + VariantCGroupMode: "v2", + VariantLayeredProduct: VariantNoValue, + VariantOS: "rhcos9", + VariantSpotCheckComponent: "etcd", + VariantSpotCheckCapability: "Scaling", + }, + }, } for _, test := range tests { t.Run(test.job, func(t *testing.T) { @@ -2545,16 +2605,16 @@ func TestAdjustJobTierBasedOnView(t *testing.T) { expectedTier: "excluded", }, { - name: "rare job is not adjusted even with non-matching variants", + name: "spotcheck job is not adjusted even with non-matching variants", variants: map[string]string{ VariantRelease: "4.22", - VariantJobTier: "rare", + VariantJobTier: "spotcheck", VariantArch: "s390x", VariantPlatform: "rosa", VariantNetwork: "sdn", VariantOwner: "chaos", }, - expectedTier: "rare", + expectedTier: "spotcheck", }, { name: "job with no release is not adjusted", diff --git a/pkg/variantregistry/snapshot.yaml b/pkg/variantregistry/snapshot.yaml index 9e5a58d2d..ecc2af2a0 100644 --- a/pkg/variantregistry/snapshot.yaml +++ b/pkg/variantregistry/snapshot.yaml @@ -257033,7 +257033,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -257047,6 +257047,8 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "12" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -257516,7 +257518,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -257530,6 +257532,8 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "12" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -257735,7 +257739,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -257749,6 +257753,8 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "12" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -258411,7 +258417,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -258425,6 +258431,8 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "12" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -258904,7 +258912,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -258918,6 +258926,8 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "13" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -258928,7 +258938,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -258942,6 +258952,8 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "13" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -259387,7 +259399,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -259401,6 +259413,8 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "13" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -259606,7 +259620,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -259620,6 +259634,8 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "13" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -260384,7 +260400,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -260398,6 +260414,8 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "13" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -260998,7 +261016,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -261012,6 +261030,8 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -261022,7 +261042,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -261036,6 +261056,8 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -261601,7 +261623,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -261615,6 +261637,8 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -261844,7 +261868,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -261858,6 +261882,8 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -262550,7 +262576,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -262564,6 +262590,8 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -263266,7 +263294,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -263280,6 +263308,8 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "15" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -263338,7 +263368,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -263352,6 +263382,8 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "15" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -263893,7 +263925,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -263907,6 +263939,8 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "15" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -264160,7 +264194,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -264174,6 +264208,8 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "15" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -264866,7 +264902,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -264880,6 +264916,8 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "15" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -265579,7 +265617,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -265593,6 +265631,8 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -265651,7 +265691,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -265665,6 +265705,8 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -266209,7 +266251,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -266223,6 +266265,8 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -266497,7 +266541,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -266511,6 +266555,8 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -267826,7 +267872,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -267840,6 +267886,8 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -268681,7 +268729,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -268695,6 +268743,8 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -268753,7 +268803,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -268767,6 +268817,8 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -269284,7 +269336,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -269298,6 +269350,8 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -269524,7 +269578,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -269538,6 +269592,8 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -270878,7 +270934,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -270892,6 +270948,8 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -271906,7 +271964,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -271920,6 +271978,8 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -271978,7 +272038,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -271992,6 +272052,8 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -272704,7 +272766,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -272718,6 +272780,8 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -273088,7 +273152,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -273102,6 +273166,8 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -275312,7 +275378,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -275326,6 +275392,8 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -276419,7 +276487,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -276433,6 +276501,8 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -276491,7 +276561,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -276505,6 +276575,8 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -277145,7 +277217,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -277159,6 +277231,8 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -277457,7 +277531,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -277471,6 +277545,8 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -279773,7 +279849,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -279787,6 +279863,8 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -280976,7 +281054,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -280990,6 +281068,8 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -281048,7 +281128,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -281062,6 +281142,8 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -281726,7 +281808,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -281740,6 +281822,8 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -282062,7 +282146,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -282076,6 +282160,8 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -284548,7 +284634,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -284562,6 +284648,8 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -285919,7 +286007,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -285933,6 +286021,8 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -285991,7 +286081,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -286005,6 +286095,8 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -286843,7 +286935,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -286857,6 +286949,8 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -287251,7 +287345,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -287265,6 +287359,8 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -290007,7 +290103,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -290021,6 +290117,8 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -291618,7 +291716,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -291632,6 +291730,8 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -291738,7 +291838,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -291752,6 +291852,8 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -292758,7 +292860,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -292772,6 +292874,8 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -293190,7 +293294,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -293204,6 +293308,8 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -296830,7 +296936,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -296844,6 +296950,8 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -298561,7 +298669,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -298575,6 +298683,8 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "23" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -298681,7 +298791,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -298695,6 +298805,8 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "23" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -299653,7 +299765,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -299667,6 +299779,8 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "23" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -300085,7 +300199,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -300099,6 +300213,8 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "23" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -303629,7 +303745,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -303643,6 +303759,8 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "23" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -305383,7 +305501,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -305397,6 +305515,8 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-cpu-partitioning: ReleaseMinor: "0" Scheduler: default SecurityMode: default + SpotCheckCapability: CPU Partitioning + SpotCheckComponent: Node Suite: unknown Topology: ha Upgrade: none @@ -305503,7 +305623,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -305517,6 +305637,8 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-etcd-scaling: ReleaseMinor: "0" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -306673,7 +306795,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -306687,6 +306809,8 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-azure-ovn-etcd-scaling: ReleaseMinor: "0" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -307105,7 +307229,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -307119,6 +307243,8 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-etcd-scaling: ReleaseMinor: "0" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -310727,7 +310853,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -310741,6 +310867,8 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-vsphere-ovn-etcd-scaling: ReleaseMinor: "0" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -342056,7 +342184,7 @@ periodic-ci-shiftstack-ci-release-4.14-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -342070,6 +342198,8 @@ periodic-ci-shiftstack-ci-release-4.14-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "14" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -342560,7 +342690,7 @@ periodic-ci-shiftstack-ci-release-4.16-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -342574,6 +342704,8 @@ periodic-ci-shiftstack-ci-release-4.16-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "16" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -342968,7 +343100,7 @@ periodic-ci-shiftstack-ci-release-4.17-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -342982,6 +343114,8 @@ periodic-ci-shiftstack-ci-release-4.17-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "17" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -343451,7 +343585,7 @@ periodic-ci-shiftstack-ci-release-4.18-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -343465,6 +343599,8 @@ periodic-ci-shiftstack-ci-release-4.18-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "18" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -343958,7 +344094,7 @@ periodic-ci-shiftstack-ci-release-4.19-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -343972,6 +344108,8 @@ periodic-ci-shiftstack-ci-release-4.19-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "19" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -344465,7 +344603,7 @@ periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -344479,6 +344617,8 @@ periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "20" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -344972,7 +345112,7 @@ periodic-ci-shiftstack-ci-release-4.21-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -344986,6 +345126,8 @@ periodic-ci-shiftstack-ci-release-4.21-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "21" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -345479,7 +345621,7 @@ periodic-ci-shiftstack-ci-release-4.22-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: rare + JobTier: spotcheck LayeredProduct: none Network: ovn NetworkAccess: default @@ -345493,6 +345635,8 @@ periodic-ci-shiftstack-ci-release-4.22-e2e-openstack-ovn-etcd-scaling: ReleaseMinor: "22" Scheduler: default SecurityMode: default + SpotCheckCapability: Scaling + SpotCheckComponent: etcd Suite: etcd-scaling Topology: ha Upgrade: none From b26cbf595f01aa3b279057193731608a347f437a Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Mon, 8 Jun 2026 11:33:44 -0300 Subject: [PATCH 14/37] Fix a bug in explanations for spot check jobs --- .../middleware/spotcheckjobs/spotcheckjobs.go | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 8f18a328c..b161ef132 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -87,20 +87,14 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, } keyStr := testKey.KeyOrDie() - atLeastOnePass := group.SuccessfulRuns >= 1 - successCount := 0 - if atLeastOnePass { - successCount = 1 - } - sampleStatus[keyStr] = crstatus.TestStatus{ TestName: syntheticTestName(group.Component, group.Capability), Component: group.Component, Capabilities: []string{group.Capability}, Variants: variantMapToSlice(group.Variants), Count: crtest.Count{ - TotalCount: 1, - SuccessCount: successCount, + TotalCount: group.TotalRuns, + SuccessCount: group.SuccessfulRuns, }, } } @@ -178,16 +172,19 @@ func (s *SpotCheckJobs) Analyze(testKey crtest.Identification, } sampleDays := int(s.reqOptions.SpotCheckSample.End.Sub(s.reqOptions.SpotCheckSample.Start).Hours() / 24) + totalRuns := testStats.SampleStats.Total() + successfulRuns := testStats.SampleStats.SuccessCount - if testStats.SampleStats.SuccessCount > 0 { + if successfulRuns > 0 { testStats.ReportStatus = crtest.NotSignificant testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("Spot-check job passed at least once in the %d-day sample window", sampleDays)) + fmt.Sprintf("Spot-check job passed %d out of %d runs in the %d-day sample window", + successfulRuns, totalRuns, sampleDays)) } else { testStats.ReportStatus = crtest.ExtremeRegression testStats.Explanations = append(testStats.Explanations, fmt.Sprintf("Spot-check job did not pass in the %d-day sample window (%d runs, 0 successes)", - sampleDays, testStats.SampleStats.Total())) + sampleDays, totalRuns)) } testStats.Comparison = crtest.SpotCheck From 2068318a36ab8ba8e2a70c024bd9dbb6e24bf26c Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Mon, 8 Jun 2026 13:54:59 -0300 Subject: [PATCH 15/37] Clarify logging --- .../middleware/spotcheckjobs/spotcheckjobs.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index b161ef132..60a84425a 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -151,7 +151,7 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup s.sampleJobDetails = map[string][]dataprovider.JobRunDetail{} } s.sampleJobDetails[testKey.KeyOrDie()] = details - s.log.Infof("loaded %d spot-check job run details for %s", len(details), testIDOpt.TestID) + s.log.WithField("variants", testIDOpt.RequestedVariants).Infof("loaded %d spot-check job run details for %s", len(details), testIDOpt.TestID) }) } } From 916f5414b4aa6e39d11c7b27d15cbc2aab977043 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Wed, 10 Jun 2026 09:31:22 -0300 Subject: [PATCH 16/37] Delay regressions until we have at least two failures for spot check jobs --- .../dataprovider/bigquery/provider.go | 6 +++- .../dataprovider/interface.go | 1 + .../middleware/spotcheckjobs/spotcheckjobs.go | 33 ++++++++++++++++--- 3 files changed, 34 insertions(+), 6 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index e52cbdd0f..01b631684 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -470,7 +470,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions %s AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, - ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names + ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names, + MAX(IF(jobs.prowjob_state != 'success', TIMESTAMP(jobs.prowjob_start), NULL)) AS last_failure FROM %s.jobs jobs %s WHERE jobs.prowjob_start >= DATETIME(@From) @@ -536,6 +537,9 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions group.JobNames = append(group.JobNames, jn.(string)) } } + if val, ok := rawRow["last_failure"]; ok && val != nil { + group.LastFailure = val.(time.Time) + } results = append(results, group) } diff --git a/pkg/api/componentreadiness/dataprovider/interface.go b/pkg/api/componentreadiness/dataprovider/interface.go index af0def7d6..e4757442a 100644 --- a/pkg/api/componentreadiness/dataprovider/interface.go +++ b/pkg/api/componentreadiness/dataprovider/interface.go @@ -116,6 +116,7 @@ type SpotCheckGroup struct { TotalRuns int `json:"total_runs"` SuccessfulRuns int `json:"successful_runs"` JobNames []string `json:"job_names"` + LastFailure time.Time `json:"last_failure"` } // JobRunDetail contains data for a single job run, used in test details drill-down. diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 60a84425a..04fd34ca4 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -96,6 +96,7 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, TotalCount: group.TotalRuns, SuccessCount: group.SuccessfulRuns, }, + LastFailure: group.LastFailure, } } @@ -161,9 +162,14 @@ func (s *SpotCheckJobs) PreAnalysis(_ crtest.Identification, return nil } -// Analyze claims spot-check tests and applies a simple heuristic: any successful run in -// the sample window means the job is healthy (NotSignificant), zero successes means -// ExtremeRegression. Returns false for non-spot-check tests to defer to other analyzers. +// Analyze claims spot-check tests and determines their status. The heuristic is: +// - Any successful run in the sample window = healthy (NotSignificant) +// - A single failed run with no successes = pending retry (MissingSample), since an +// external component will trigger a retry for failed spot-check jobs +// - Two or more failed runs with no successes = confirmed regression (ExtremeRegression) +// - No runs at all = no data (MissingSample) +// +// Returns false for non-spot-check tests to defer to other analyzers. func (s *SpotCheckJobs) Analyze(testKey crtest.Identification, testStats *testdetails.TestComparison) (bool, error) { @@ -174,17 +180,34 @@ func (s *SpotCheckJobs) Analyze(testKey crtest.Identification, sampleDays := int(s.reqOptions.SpotCheckSample.End.Sub(s.reqOptions.SpotCheckSample.Start).Hours() / 24) totalRuns := testStats.SampleStats.Total() successfulRuns := testStats.SampleStats.SuccessCount + failedRuns := totalRuns - successfulRuns - if successfulRuns > 0 { + switch { + case successfulRuns > 0: testStats.ReportStatus = crtest.NotSignificant testStats.Explanations = append(testStats.Explanations, fmt.Sprintf("Spot-check job passed %d out of %d runs in the %d-day sample window", successfulRuns, totalRuns, sampleDays)) - } else { + case failedRuns >= 3: testStats.ReportStatus = crtest.ExtremeRegression testStats.Explanations = append(testStats.Explanations, fmt.Sprintf("Spot-check job did not pass in the %d-day sample window (%d runs, 0 successes)", sampleDays, totalRuns)) + case failedRuns == 2: + testStats.ReportStatus = crtest.SignificantRegression + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Spot-check job failed %d times in the %d-day sample window with no successes", + failedRuns, sampleDays)) + case failedRuns == 1: + testStats.ReportStatus = crtest.MissingSample + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("Spot-check job failed once in the %d-day sample window; awaiting retry before flagging regression", + sampleDays)) + default: + // No runs at all + testStats.ReportStatus = crtest.MissingSample + testStats.Explanations = append(testStats.Explanations, + fmt.Sprintf("No spot-check job runs found in the %d-day sample window", sampleDays)) } testStats.Comparison = crtest.SpotCheck From 39fc86395439f4a05cc8745b423c0f78e07b8029 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 08:36:33 -0300 Subject: [PATCH 17/37] Add spot check job middleware unit tests --- .../spotcheckjobs/spotcheckjobs_test.go | 156 ++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go new file mode 100644 index 000000000..dd05c8ea6 --- /dev/null +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go @@ -0,0 +1,156 @@ +package spotcheckjobs + +import ( + "testing" + "time" + + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestAnalyze(t *testing.T) { + now := time.Now() + sampleStart := now.Add(-30 * 24 * time.Hour) + sampleEnd := now + + spotCheckSample := reqopts.Release{ + Start: sampleStart, + End: sampleEnd, + } + + mw := &SpotCheckJobs{ + reqOptions: reqopts.RequestOptions{ + SpotCheckSample: &spotCheckSample, + }, + } + + spotCheckTestKey := crtest.Identification{ + RowIdentification: crtest.RowIdentification{ + TestID: "spotcheck:etcd:scaling", + }, + } + + nonSpotCheckTestKey := crtest.Identification{ + RowIdentification: crtest.RowIdentification{ + TestID: "openshift-tests:some-real-test", + }, + } + + tests := []struct { + name string + testKey crtest.Identification + successCount int + failureCount int + expectedStatus crtest.Status + expectHandled bool + explanationContains string + }{ + { + name: "non-spot-check test is not handled", + testKey: nonSpotCheckTestKey, + successCount: 0, + failureCount: 3, + expectedStatus: 0, // unchanged + expectHandled: false, + }, + { + name: "no runs - MissingSample", + testKey: spotCheckTestKey, + successCount: 0, + failureCount: 0, + expectedStatus: crtest.MissingSample, + expectHandled: true, + explanationContains: "No spot-check job runs found", + }, + { + name: "1 failure 0 successes - MissingSample awaiting retry", + testKey: spotCheckTestKey, + successCount: 0, + failureCount: 1, + expectedStatus: crtest.MissingSample, + expectHandled: true, + explanationContains: "awaiting retry", + }, + { + name: "2 failures 0 successes - SignificantRegression", + testKey: spotCheckTestKey, + successCount: 0, + failureCount: 2, + expectedStatus: crtest.SignificantRegression, + expectHandled: true, + explanationContains: "failed 2 times", + }, + { + name: "3 failures 0 successes - ExtremeRegression", + testKey: spotCheckTestKey, + successCount: 0, + failureCount: 3, + expectedStatus: crtest.ExtremeRegression, + expectHandled: true, + explanationContains: "did not pass", + }, + { + name: "5 failures 0 successes - ExtremeRegression", + testKey: spotCheckTestKey, + successCount: 0, + failureCount: 5, + expectedStatus: crtest.ExtremeRegression, + expectHandled: true, + explanationContains: "5 runs, 0 successes", + }, + { + name: "1 success 0 failures - NotSignificant", + testKey: spotCheckTestKey, + successCount: 1, + failureCount: 0, + expectedStatus: crtest.NotSignificant, + expectHandled: true, + explanationContains: "passed 1 out of 1", + }, + { + name: "1 success 3 failures - NotSignificant", + testKey: spotCheckTestKey, + successCount: 1, + failureCount: 3, + expectedStatus: crtest.NotSignificant, + expectHandled: true, + explanationContains: "passed 1 out of 4", + }, + { + name: "4 successes 0 failures - NotSignificant", + testKey: spotCheckTestKey, + successCount: 4, + failureCount: 0, + expectedStatus: crtest.NotSignificant, + expectHandled: true, + explanationContains: "passed 4 out of 4", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + testStats := &testdetails.TestComparison{ + SampleStats: testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(tt.successCount, tt.failureCount, 0, false), + }, + } + + handled, err := mw.Analyze(tt.testKey, testStats) + require.NoError(t, err) + assert.Equal(t, tt.expectHandled, handled) + + if !tt.expectHandled { + return + } + + assert.Equal(t, tt.expectedStatus, testStats.ReportStatus, "unexpected status") + assert.Equal(t, crtest.SpotCheck, testStats.Comparison) + assert.Nil(t, testStats.BaseStats) + require.Len(t, testStats.Explanations, 1) + assert.Contains(t, testStats.Explanations[0], tt.explanationContains) + }) + } +} From 0985309b8d46db4df86f4b7ddfa427f82d3dc13c Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 09:00:26 -0300 Subject: [PATCH 18/37] Fix bug with non-existant components --- pkg/api/componentreadiness/dataprovider/bigquery/provider.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 01b631684..ca683d47f 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -382,8 +382,8 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // syncer has run and all spot-check jobs have the new variants. const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, CASE - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node' - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'etcd' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node / Kubelet' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Etcd' END)` const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, From 16a079940d543456dafe8a80fe55e322c4b4f228 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 10:19:16 -0300 Subject: [PATCH 19/37] Fix missing spotcheck regressions on component drilldown --- pkg/api/componentreadiness/utils/queryparamparser.go | 10 ++++++++++ sippy-ng/src/component_readiness/CompReadyUtils.js | 4 ++++ 2 files changed, 14 insertions(+) diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index 60caa9662..61acbbaf1 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -224,6 +224,16 @@ func ParseComponentReportRequest( } } + // If no view provided SpotCheckSample, default it from the sample release dates + // so spot-check middleware runs on drill-down requests too. + if opts.SpotCheckSample == nil && !opts.SampleRelease.Start.IsZero() && !opts.SampleRelease.End.IsZero() { + opts.SpotCheckSample = &reqopts.Release{ + Name: opts.SampleRelease.Name, + Start: opts.SampleRelease.Start, + End: opts.SampleRelease.End, + } + } + opts.CacheOption = cache.NewStandardCROptions(crTimeRoundingFactor, crTimeRoundingOffset) opts.CacheOption.ForceRefresh, err = ParseBoolArg(req, "forceRefresh", false) if err != nil { diff --git a/sippy-ng/src/component_readiness/CompReadyUtils.js b/sippy-ng/src/component_readiness/CompReadyUtils.js index 88a5ee14f..cfeb6610f 100644 --- a/sippy-ng/src/component_readiness/CompReadyUtils.js +++ b/sippy-ng/src/component_readiness/CompReadyUtils.js @@ -386,6 +386,10 @@ export function getUpdatedUrlParts(vars) { //component: vars.component, } + if (vars.view) { + valuesMap.view = vars.view + } + if (vars.samplePROrg && vars.samplePRRepo && vars.samplePRNumber) { valuesMap.samplePROrg = vars.samplePROrg valuesMap.samplePRRepo = vars.samplePRRepo From 05bf3f0ed479a5774177599b96cb34a1dac5e2ea Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 11:25:53 -0300 Subject: [PATCH 20/37] Fix environment filtering for spot check --- .../middleware/spotcheckjobs/spotcheckjobs.go | 17 ++ .../spotcheckjobs/spotcheckjobs_test.go | 173 ++++++++++++++++++ 2 files changed, 190 insertions(+) diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index 04fd34ca4..c9c195441 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -75,12 +75,20 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, } sampleStatus := map[string]crstatus.TestStatus{} + requestedVariants := map[string]string{} + if len(s.reqOptions.TestIDOptions) > 0 { + requestedVariants = s.reqOptions.TestIDOptions[0].RequestedVariants + } for _, group := range groups { if group.Component == "" || group.Capability == "" { s.log.Warnf("skipping spot-check group with empty component/capability: %+v", group) continue } + if !variantsMatch(group.Variants, requestedVariants) { + continue + } + testKey := crtest.KeyWithVariants{ TestID: syntheticTestID(group.Component, group.Capability), Variants: group.Variants, @@ -303,6 +311,15 @@ func syntheticTestNameFromID(testID string) string { return testID } +func variantsMatch(groupVariants, requestedVariants map[string]string) bool { + for k, v := range requestedVariants { + if gv, ok := groupVariants[k]; !ok || gv != v { + return false + } + } + return true +} + func variantMapToSlice(m map[string]string) []string { result := make([]string, 0, len(m)) for k, v := range m { diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go index dd05c8ea6..6f49cdc91 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go @@ -1,9 +1,16 @@ package spotcheckjobs import ( + "context" + "strings" + "sync" "testing" "time" + log "github.com/sirupsen/logrus" + + "github.com/openshift/sippy/pkg/api/componentreadiness/dataprovider" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crstatus" "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" @@ -11,6 +18,172 @@ import ( "github.com/stretchr/testify/require" ) +// stubProvider returns a fixed set of SpotCheckGroups; all other methods panic. +type stubProvider struct { + dataprovider.DataProvider // embed to satisfy the interface; only spot-check methods are implemented + groups []dataprovider.SpotCheckGroup +} + +func (s *stubProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, + _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + return s.groups, nil +} + +func (s *stubProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, + _ crtest.JobVariants, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + return nil, nil +} + +// runQuery is a test helper that runs the middleware Query and collects injected sample statuses. +func runQuery(t *testing.T, mw *SpotCheckJobs) map[string]crstatus.TestStatus { + t.Helper() + wg := &sync.WaitGroup{} + baseCh := make(chan map[string]crstatus.TestStatus, 1) + sampleCh := make(chan map[string]crstatus.TestStatus, 10) + errCh := make(chan error, 10) + + mw.Query(context.Background(), wg, crtest.JobVariants{}, baseCh, sampleCh, errCh) + wg.Wait() + close(sampleCh) + close(errCh) + + for err := range errCh { + t.Fatalf("unexpected error from Query: %v", err) + } + + merged := map[string]crstatus.TestStatus{} + for batch := range sampleCh { + for k, v := range batch { + merged[k] = v + } + } + return merged +} + +func TestVariantsMatch(t *testing.T) { + tests := []struct { + name string + groupVariants map[string]string + requestedVariants map[string]string + expected bool + }{ + { + name: "empty requested matches everything", + groupVariants: map[string]string{"Network": "ovn", "Platform": "aws"}, + requestedVariants: map[string]string{}, + expected: true, + }, + { + name: "nil requested matches everything", + groupVariants: map[string]string{"Network": "ovn", "Platform": "aws"}, + requestedVariants: nil, + expected: true, + }, + { + name: "single match", + groupVariants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, + requestedVariants: map[string]string{"Platform": "aws"}, + expected: true, + }, + { + name: "all match", + groupVariants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, + requestedVariants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, + expected: true, + }, + { + name: "one mismatch", + groupVariants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, + requestedVariants: map[string]string{"Network": "ovn", "Platform": "gcp"}, + expected: false, + }, + { + name: "key missing from group", + groupVariants: map[string]string{"Network": "ovn"}, + requestedVariants: map[string]string{"Platform": "aws"}, + expected: false, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + assert.Equal(t, tt.expected, variantsMatch(tt.groupVariants, tt.requestedVariants)) + }) + } +} + +func TestQueryFiltering(t *testing.T) { + spotCheckSample := reqopts.Release{Start: time.Now().Add(-30 * 24 * time.Hour), End: time.Now()} + + allGroups := []dataprovider.SpotCheckGroup{ + {Component: "Etcd", Capability: "Scaling", Variants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, TotalRuns: 3, SuccessfulRuns: 2}, + {Component: "Etcd", Capability: "Scaling", Variants: map[string]string{"Network": "ovn", "Platform": "gcp", "Topology": "ha"}, TotalRuns: 3, SuccessfulRuns: 3}, + {Component: "Node / Kubelet", Capability: "CPU Partitioning", Variants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, TotalRuns: 2, SuccessfulRuns: 1}, + } + + tests := []struct { + name string + requestedVariants map[string]string + requestedComp string + expectedKeys []string // syntheticTestID substrings to expect + notExpectedKeys []string + }{ + { + name: "no filter - all groups injected", + requestedVariants: map[string]string{}, + expectedKeys: []string{"spotcheck:etcd:scaling", "spotcheck:node / kubelet:cpu-partitioning"}, + }, + { + name: "environment filter Platform=aws - excludes gcp", + requestedVariants: map[string]string{"Platform": "aws"}, + expectedKeys: []string{"spotcheck:etcd:scaling", "spotcheck:node / kubelet:cpu-partitioning"}, + notExpectedKeys: []string{}, + }, + { + name: "environment filter Platform=gcp - only gcp etcd", + requestedVariants: map[string]string{"Platform": "gcp"}, + notExpectedKeys: []string{"spotcheck:node / kubelet:cpu-partitioning"}, + }, + { + name: "environment filter Platform=metal - nothing matches", + requestedVariants: map[string]string{"Platform": "metal"}, + expectedKeys: []string{}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mw := &SpotCheckJobs{ + dataProvider: &stubProvider{groups: allGroups}, + reqOptions: reqopts.RequestOptions{ + SpotCheckSample: &spotCheckSample, + TestIDOptions: []reqopts.TestIdentification{ + {Component: tt.requestedComp, RequestedVariants: tt.requestedVariants}, + }, + }, + log: log.WithField("test", tt.name), + } + + result := runQuery(t, mw) + + for _, key := range tt.expectedKeys { + found := false + for k := range result { + if strings.Contains(k, key) { + found = true + break + } + } + assert.True(t, found, "expected key containing %q in results", key) + } + for _, key := range tt.notExpectedKeys { + for k := range result { + assert.False(t, strings.Contains(k, key), "did not expect key containing %q in results", key) + } + } + }) + } +} + func TestAnalyze(t *testing.T) { now := time.Now() sampleStart := now.Add(-30 * 24 * time.Hour) From 47fa25e9405f4802707f9bac81c8c7f7698a2679 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 14:54:05 -0300 Subject: [PATCH 21/37] Fix spotcheck components in variant registry --- pkg/variantregistry/ocp.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/variantregistry/ocp.go b/pkg/variantregistry/ocp.go index ddc786051..5f286ca7a 100644 --- a/pkg/variantregistry/ocp.go +++ b/pkg/variantregistry/ocp.go @@ -742,8 +742,8 @@ func setSpotCheckClassification(_ logrus.FieldLogger, variants map[string]string component string capability string }{ - {[]string{"-cpu-partitioning"}, "Node", "CPU Partitioning"}, - {[]string{"-etcd-scaling"}, "etcd", "Scaling"}, + {[]string{"-cpu-partitioning"}, "Node / kubelet", "CPU Partitioning"}, + {[]string{"-etcd-scaling"}, "Etcd", "Scaling"}, } for _, p := range spotCheckPatterns { From c973cf9ccb24e0c5a5df1b924b417a4f6b270e99 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 14:59:49 -0300 Subject: [PATCH 22/37] Ensure spotcheck jobs have a Component/Capability --- pkg/variantregistry/ocp.go | 23 +++++++++++++++++++++++ pkg/variantregistry/snapshot.go | 12 +++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/pkg/variantregistry/ocp.go b/pkg/variantregistry/ocp.go index 5f286ca7a..113677621 100644 --- a/pkg/variantregistry/ocp.go +++ b/pkg/variantregistry/ocp.go @@ -260,9 +260,30 @@ ORDER BY j.prowjob_job_name; dur := time.Since(start) log.WithField("count", count.Load()).Infof("processed primary job list in %s", dur) + var errs []string + for jobName, variants := range variantsByJob { + if err := validateSpotCheckVariants(jobName, variants); err != nil { + errs = append(errs, err.Error()) + } + } + if len(errs) > 0 { + return nil, errors.New("variant registry validation failed:\n" + strings.Join(errs, "\n")) + } + return variantsByJob, nil } +// validateSpotCheckVariants returns an error if a job has JobTier=spotcheck without both +// SpotCheckComponent and SpotCheckCapability defined. +func validateSpotCheckVariants(jobName string, variants map[string]string) error { + if variants[VariantJobTier] == "spotcheck" { + if variants[VariantSpotCheckComponent] == "" || variants[VariantSpotCheckCapability] == "" { + return fmt.Errorf("job %q has JobTier=spotcheck but is missing SpotCheckComponent or SpotCheckCapability", jobName) + } + } + return nil +} + // fileVariantsToIgnore are values in the cluster-data.json that vary by run, and are not consistent for the job itself. // These are unsuited for variants. var fileVariantsToIgnore = map[string]bool{ @@ -734,6 +755,8 @@ func (v *OCPVariantLoader) setRelease(logger logrus.FieldLogger, variants map[st // assessed on a simple pass/fail basis rather than per-test-case junit analysis. // The SpotCheckComponent and SpotCheckCapability variants control where these synthetic // results appear in the component readiness report. +// +// Be sure to use real Component names from OCPBUGS. func setSpotCheckClassification(_ logrus.FieldLogger, variants map[string]string, jobName string) { jobNameLower := strings.ToLower(jobName) diff --git a/pkg/variantregistry/snapshot.go b/pkg/variantregistry/snapshot.go index 816cf296e..614dc7bf1 100644 --- a/pkg/variantregistry/snapshot.go +++ b/pkg/variantregistry/snapshot.go @@ -1,6 +1,7 @@ package variantregistry import ( + "fmt" "os" "strings" @@ -34,6 +35,7 @@ func NewVariantSnapshot(config *v1.SippyConfig, views []crview.View, syntheticRe func (s *VariantSnapshot) Identify() (JobVariants, error) { newVariants := map[string]map[string]string{} variantSyncer := OCPVariantLoader{config: s.config, views: s.views, syntheticReleaseJobOverrides: s.syntheticReleaseJobOverrides} + var errs []string for _, releaseCfg := range s.config.Releases { for job := range releaseCfg.Jobs { if isIgnoredJob(job) { @@ -42,10 +44,18 @@ func (s *VariantSnapshot) Identify() (JobVariants, error) { if _, done := newVariants[job]; done { continue } - newVariants[job] = variantSyncer.CalculateVariantsForJob(s.log, job, nil) + variants := variantSyncer.CalculateVariantsForJob(s.log, job, nil) + newVariants[job] = variants + if err := validateSpotCheckVariants(job, variants); err != nil { + errs = append(errs, err.Error()) + } } } + if len(errs) > 0 { + return nil, fmt.Errorf("variant registry validation failed:\n%s", strings.Join(errs, "\n")) + } + return newVariants, nil } From 2355f9e3f147b618493b5a0741b0852acf92d0f0 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Thu, 11 Jun 2026 14:59:56 -0300 Subject: [PATCH 23/37] Variant snapshot update --- pkg/variantregistry/snapshot.yaml | 144 +++++++++++++++--------------- 1 file changed, 72 insertions(+), 72 deletions(-) diff --git a/pkg/variantregistry/snapshot.yaml b/pkg/variantregistry/snapshot.yaml index ecc2af2a0..074d746d5 100644 --- a/pkg/variantregistry/snapshot.yaml +++ b/pkg/variantregistry/snapshot.yaml @@ -257048,7 +257048,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -257533,7 +257533,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -257754,7 +257754,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -258432,7 +258432,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -258927,7 +258927,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -258953,7 +258953,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -259414,7 +259414,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -259635,7 +259635,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -260415,7 +260415,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -261031,7 +261031,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -261057,7 +261057,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -261638,7 +261638,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -261883,7 +261883,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -262591,7 +262591,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -263309,7 +263309,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -263383,7 +263383,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -263940,7 +263940,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -264209,7 +264209,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -264917,7 +264917,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -265632,7 +265632,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -265706,7 +265706,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -266266,7 +266266,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -266556,7 +266556,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -267887,7 +267887,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -268744,7 +268744,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -268818,7 +268818,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -269351,7 +269351,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -269593,7 +269593,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -270949,7 +270949,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -271979,7 +271979,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -272053,7 +272053,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -272781,7 +272781,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -273167,7 +273167,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -275393,7 +275393,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -276502,7 +276502,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -276576,7 +276576,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -277232,7 +277232,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -277546,7 +277546,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -279864,7 +279864,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -281069,7 +281069,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -281143,7 +281143,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -281823,7 +281823,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -282161,7 +282161,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -284649,7 +284649,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -286022,7 +286022,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -286096,7 +286096,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -286950,7 +286950,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -287360,7 +287360,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -290118,7 +290118,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -291731,7 +291731,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -291853,7 +291853,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -292875,7 +292875,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -293309,7 +293309,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -296951,7 +296951,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -298684,7 +298684,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -298806,7 +298806,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -299780,7 +299780,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -300214,7 +300214,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -303760,7 +303760,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -305516,7 +305516,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-cpu-partitioning: Scheduler: default SecurityMode: default SpotCheckCapability: CPU Partitioning - SpotCheckComponent: Node + SpotCheckComponent: Node / kubelet Suite: unknown Topology: ha Upgrade: none @@ -305638,7 +305638,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -306810,7 +306810,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-azure-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -307244,7 +307244,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -310868,7 +310868,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-vsphere-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -342199,7 +342199,7 @@ periodic-ci-shiftstack-ci-release-4.14-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -342705,7 +342705,7 @@ periodic-ci-shiftstack-ci-release-4.16-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -343115,7 +343115,7 @@ periodic-ci-shiftstack-ci-release-4.17-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -343600,7 +343600,7 @@ periodic-ci-shiftstack-ci-release-4.18-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -344109,7 +344109,7 @@ periodic-ci-shiftstack-ci-release-4.19-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -344618,7 +344618,7 @@ periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -345127,7 +345127,7 @@ periodic-ci-shiftstack-ci-release-4.21-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none @@ -345636,7 +345636,7 @@ periodic-ci-shiftstack-ci-release-4.22-e2e-openstack-ovn-etcd-scaling: Scheduler: default SecurityMode: default SpotCheckCapability: Scaling - SpotCheckComponent: etcd + SpotCheckComponent: Etcd Suite: etcd-scaling Topology: ha Upgrade: none From 6ac35ffecf88b7b245e376947e281d072a384b1e Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 07:48:37 -0300 Subject: [PATCH 24/37] Fix test data mismatch --- pkg/variantregistry/ocp_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/variantregistry/ocp_test.go b/pkg/variantregistry/ocp_test.go index 219b2701d..6859861a4 100644 --- a/pkg/variantregistry/ocp_test.go +++ b/pkg/variantregistry/ocp_test.go @@ -2202,7 +2202,7 @@ func TestVariantSyncer(t *testing.T) { VariantCGroupMode: "v2", VariantLayeredProduct: VariantNoValue, VariantOS: "rhcos9", - VariantSpotCheckComponent: "Node", + VariantSpotCheckComponent: "Node / kubelet", VariantSpotCheckCapability: "CPU Partitioning", }, }, @@ -2232,7 +2232,7 @@ func TestVariantSyncer(t *testing.T) { VariantCGroupMode: "v2", VariantLayeredProduct: VariantNoValue, VariantOS: "rhcos9", - VariantSpotCheckComponent: "etcd", + VariantSpotCheckComponent: "Etcd", VariantSpotCheckCapability: "Scaling", }, }, From c98c197dc3601c970c44ff72df3798a93de7eebb Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 07:49:46 -0300 Subject: [PATCH 25/37] AI generated test coverage for middleware analyses functions --- .../alltestspassrate/alltestspassrate_test.go | 87 ++++++++ .../fisherexact/fisherexact_test.go | 209 ++++++++++++++++++ .../newtestpassrate/newtestpassrate_test.go | 106 +++++++++ 3 files changed, 402 insertions(+) create mode 100644 pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate_test.go create mode 100644 pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go create mode 100644 pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate_test.go diff --git a/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate_test.go b/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate_test.go new file mode 100644 index 000000000..58eecb9bf --- /dev/null +++ b/pkg/api/componentreadiness/middleware/alltestspassrate/alltestspassrate_test.go @@ -0,0 +1,87 @@ +package alltestspassrate + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/analysis" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +func TestAllTestsPassRateAnalyze(t *testing.T) { + tests := []struct { + name string + passRateRequiredAll int + minimumFailure int + successCount int + failureCount int + expectHandled bool + expectRegressionStatus bool + }{ + { + name: "PassRateRequiredAllTests=0 returns not handled, testStats unchanged", + passRateRequiredAll: 0, + successCount: 3, + failureCount: 3, + expectHandled: false, + }, + { + name: "PassRateRequiredAllTests>0 with passing run returns handled and NotSignificant", + passRateRequiredAll: 95, + successCount: 10, + failureCount: 0, + expectHandled: true, + expectRegressionStatus: false, + }, + { + name: "PassRateRequiredAllTests>0 with failing run returns handled and regression", + passRateRequiredAll: 95, + minimumFailure: 1, + successCount: 93, + failureCount: 7, + expectHandled: true, + expectRegressionStatus: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mw := NewAllTestsPassRateMiddleware(reqopts.RequestOptions{ + AdvancedOption: reqopts.Advanced{ + PassRateRequiredAllTests: tt.passRateRequiredAll, + MinimumFailure: tt.minimumFailure, + }, + }) + + testStats := &testdetails.TestComparison{ + SampleStats: testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(tt.successCount, tt.failureCount, 0, false), + }, + } + + handled, err := mw.Analyze(crtest.Identification{}, testStats) + require.NoError(t, err) + assert.Equal(t, tt.expectHandled, handled) + + if !tt.expectHandled { + // testStats must not be modified + assert.Equal(t, crtest.Status(0), testStats.ReportStatus) + assert.Empty(t, testStats.Explanations) + return + } + + // analysis.BuildPassRateTestStats must have been applied + if tt.expectRegressionStatus { + assert.Less(t, int(testStats.ReportStatus), 0, "expected a regression status (negative)") + assert.Equal(t, crtest.PassRate, testStats.Comparison) + } else { + assert.Equal(t, crtest.NotSignificant, testStats.ReportStatus) + assert.Contains(t, testStats.Explanations, analysis.ExplanationNoRegression) + } + }) + } +} diff --git a/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go new file mode 100644 index 000000000..0a2cf998a --- /dev/null +++ b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go @@ -0,0 +1,209 @@ +package fisherexact + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/openshift/sippy/pkg/api/componentreadiness/middleware/analysis" + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +// baseStats constructs a *testdetails.ReleaseStats with the given pass/fail counts. +func baseStats(success, failure int) *testdetails.ReleaseStats { + return &testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(success, failure, 0, false), + } +} + +// sampleStats constructs a testdetails.ReleaseStats with the given pass/fail counts. +func sampleStats(success, failure int) testdetails.ReleaseStats { + return testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(success, failure, 0, false), + } +} + +func TestFisherExactAnalyze(t *testing.T) { + tests := []struct { + name string + opts reqopts.Advanced + inputStats *testdetails.TestComparison + wantHandled bool + wantStatus crtest.Status + wantFisherZero bool // true: assert FisherExact == 0.0 + wantExplanations int // expected len(Explanations); -1 means skip check + explanationContains string + wantConfidence int // if non-zero, assert testStats.RequiredConfidence == this after call + }{ + { + name: "sample total=0, IgnoreMissing=false → MissingSample", + opts: reqopts.Advanced{IgnoreMissing: false, Confidence: 95}, + inputStats: &testdetails.TestComparison{ + SampleStats: sampleStats(0, 0), + BaseStats: baseStats(900, 100), + }, + wantHandled: true, + wantStatus: crtest.MissingSample, + wantFisherZero: true, + wantExplanations: 1, + explanationContains: analysis.ExplanationNoRegression, + }, + { + name: "sample total=0, IgnoreMissing=true → NotSignificant", + opts: reqopts.Advanced{IgnoreMissing: true, Confidence: 95}, + inputStats: &testdetails.TestComparison{ + SampleStats: sampleStats(0, 0), + BaseStats: baseStats(900, 100), + }, + wantHandled: true, + wantStatus: crtest.NotSignificant, + wantFisherZero: true, + wantExplanations: 1, + explanationContains: analysis.ExplanationNoRegression, + }, + { + name: "minimum-failure triage: failures below threshold → NotSignificant, no Fisher", + // sample has 5 failures, MinimumFailure=10 → early return + opts: reqopts.Advanced{Confidence: 95, MinimumFailure: 10, PityFactor: 5}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 95, + SampleStats: sampleStats(95, 5), + BaseStats: baseStats(80, 20), + }, + wantHandled: true, + wantStatus: crtest.NotSignificant, + wantFisherZero: true, + wantExplanations: 0, + }, + { + name: "improved performance, Fisher significant → SignificantImprovement", + // base 50%, sample 90%: massive improvement, Fisher will be significant at 95% + opts: reqopts.Advanced{Confidence: 95}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 95, + SampleStats: sampleStats(900, 100), + BaseStats: baseStats(500, 500), + }, + wantHandled: true, + wantStatus: crtest.SignificantImprovement, + wantFisherZero: false, + wantExplanations: 0, + }, + { + name: "degraded within pity → NotSignificant, no Fisher call", + // base 95%, sample 93%: drop 2% < PityFactor 5% + PityAdjustment 1% = 6% + opts: reqopts.Advanced{Confidence: 95, PityFactor: 5}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 95, + PityAdjustment: 1.0, + SampleStats: sampleStats(93, 7), + BaseStats: baseStats(95, 5), + }, + wantHandled: true, + wantStatus: crtest.NotSignificant, + wantFisherZero: false, // FisherExact set to the computed (zero) val at end + wantExplanations: 0, + }, + { + name: "degraded beyond pity, drop 13% → SignificantRegression", + // base 95%, sample 82%: drop 13% < 15%, Fisher significant at 95% + opts: reqopts.Advanced{Confidence: 95, PityFactor: 5}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 95, + SampleStats: sampleStats(820, 180), + BaseStats: baseStats(950, 50), + }, + wantHandled: true, + wantStatus: crtest.SignificantRegression, + wantFisherZero: false, + wantExplanations: 3, + explanationContains: "Significant", + }, + { + name: "degraded beyond pity, drop 25% → ExtremeRegression", + // base 95%, sample 70%: drop 25% > 15%, Fisher significant at 95% + opts: reqopts.Advanced{Confidence: 95, PityFactor: 5}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 95, + SampleStats: sampleStats(700, 300), + BaseStats: baseStats(950, 50), + }, + wantHandled: true, + wantStatus: crtest.ExtremeRegression, + wantFisherZero: false, + wantExplanations: 3, + explanationContains: "Extreme", + }, + { + name: "RequiredConfidence=0 inherits opts.Confidence", + // Use missing-sample to keep the outcome predictable; focus is the confidence field. + opts: reqopts.Advanced{IgnoreMissing: true, Confidence: 97}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 0, + SampleStats: sampleStats(0, 0), + }, + wantHandled: true, + wantStatus: crtest.NotSignificant, + wantFisherZero: true, + wantExplanations: 1, // missing-sample path always appends ExplanationNoRegression + wantConfidence: 97, + }, + { + name: "RequiredConfidence pre-set is not overwritten by opts.Confidence", + opts: reqopts.Advanced{IgnoreMissing: true, Confidence: 95}, + inputStats: &testdetails.TestComparison{ + RequiredConfidence: 90, + SampleStats: sampleStats(0, 0), + }, + wantHandled: true, + wantStatus: crtest.NotSignificant, + wantFisherZero: true, + wantExplanations: 1, + wantConfidence: 90, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mw := NewFisherExactMiddleware(reqopts.RequestOptions{ + AdvancedOption: tt.opts, + }) + + handled, err := mw.Analyze(crtest.Identification{}, tt.inputStats) + require.NoError(t, err) + assert.Equal(t, tt.wantHandled, handled) + assert.Equal(t, tt.wantStatus, tt.inputStats.ReportStatus) + + require.NotNil(t, tt.inputStats.FisherExact) + if tt.wantFisherZero { + assert.Equal(t, 0.0, *tt.inputStats.FisherExact) + } else { + assert.GreaterOrEqual(t, *tt.inputStats.FisherExact, 0.0) + } + + if tt.wantExplanations >= 0 { + assert.Len(t, tt.inputStats.Explanations, tt.wantExplanations) + } + if tt.explanationContains != "" { + assert.True(t, func() bool { + for _, e := range tt.inputStats.Explanations { + if len(e) >= len(tt.explanationContains) { + for i := 0; i <= len(e)-len(tt.explanationContains); i++ { + if e[i:i+len(tt.explanationContains)] == tt.explanationContains { + return true + } + } + } + } + return false + }(), "expected an explanation containing %q, got: %v", tt.explanationContains, tt.inputStats.Explanations) + } + if tt.wantConfidence != 0 { + assert.Equal(t, tt.wantConfidence, tt.inputStats.RequiredConfidence) + } + }) + } +} diff --git a/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate_test.go b/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate_test.go new file mode 100644 index 000000000..354420062 --- /dev/null +++ b/pkg/api/componentreadiness/middleware/newtestpassrate/newtestpassrate_test.go @@ -0,0 +1,106 @@ +package newtestpassrate + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/openshift/sippy/pkg/apis/api/componentreport/crtest" + "github.com/openshift/sippy/pkg/apis/api/componentreport/reqopts" + "github.com/openshift/sippy/pkg/apis/api/componentreport/testdetails" +) + +func TestNewTestPassRateAnalyze(t *testing.T) { + existingBase := &testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(80, 20, 0, false), + } + + tests := []struct { + name string + passRateNewTests int + passRateAllTests int + baseStats *testdetails.ReleaseStats + sampleSuccess int + sampleFailure int + wantHandled bool + wantStatus crtest.Status + }{ + { + name: "PassRateRequiredNewTests=0 → not handled", + passRateNewTests: 0, + baseStats: nil, + sampleSuccess: 5, + sampleFailure: 5, + wantHandled: false, + wantStatus: 0, // unchanged zero value + }, + { + name: "BaseStats non-nil with runs → not handled", + passRateNewTests: 95, + baseStats: existingBase, + sampleSuccess: 8, + sampleFailure: 2, + wantHandled: false, + wantStatus: 0, + }, + { + name: "new test, passing sample, PassRateRequiredAllTests=0 → MissingBasis", + // 10 successes, 0 failures → 100% pass rate > 95% required → NotSignificant from + // BuildPassRateTestStats, then overridden to MissingBasis because PassRateRequiredAllTests==0. + passRateNewTests: 95, + passRateAllTests: 0, + baseStats: nil, + sampleSuccess: 10, + sampleFailure: 0, + wantHandled: true, + wantStatus: crtest.MissingBasis, + }, + { + name: "new test, passing sample, PassRateRequiredAllTests>0 → NotSignificant", + // Same passing stats but PassRateRequiredAllTests is set, so MissingBasis override is skipped. + passRateNewTests: 95, + passRateAllTests: 90, + baseStats: nil, + sampleSuccess: 10, + sampleFailure: 0, + wantHandled: true, + wantStatus: crtest.NotSignificant, + }, + { + name: "new test, failing sample → regression status", + // 3 successes, 4 failures: Total=7 (≥6), pass rate ≈42.8% < 95% required, + // drop > allowed → ExtremeRegression from BuildPassRateTestStats. + passRateNewTests: 95, + passRateAllTests: 0, + baseStats: nil, + sampleSuccess: 3, + sampleFailure: 4, + wantHandled: true, + wantStatus: crtest.ExtremeRegression, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mw := NewNewTestPassRateMiddleware(reqopts.RequestOptions{ + AdvancedOption: reqopts.Advanced{ + PassRateRequiredNewTests: tt.passRateNewTests, + PassRateRequiredAllTests: tt.passRateAllTests, + }, + }) + + testStats := &testdetails.TestComparison{ + BaseStats: tt.baseStats, + SampleStats: testdetails.ReleaseStats{ + Stats: crtest.NewTestStats(tt.sampleSuccess, tt.sampleFailure, 0, false), + }, + } + + handled, err := mw.Analyze(crtest.Identification{}, testStats) + require.NoError(t, err) + assert.Equal(t, tt.wantHandled, handled) + assert.Equal(t, tt.wantStatus, testStats.ReportStatus) + }) + } +} From 0b71bbed5549bbdb8c3d784a3fefd7ab7e641f76 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 08:06:38 -0300 Subject: [PATCH 26/37] Cleanup test code --- .../middleware/fisherexact/fisherexact_test.go | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go index 0a2cf998a..1e30beb76 100644 --- a/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go +++ b/pkg/api/componentreadiness/middleware/fisherexact/fisherexact_test.go @@ -1,6 +1,7 @@ package fisherexact import ( + "strings" "testing" "github.com/stretchr/testify/assert" @@ -190,12 +191,8 @@ func TestFisherExactAnalyze(t *testing.T) { if tt.explanationContains != "" { assert.True(t, func() bool { for _, e := range tt.inputStats.Explanations { - if len(e) >= len(tt.explanationContains) { - for i := 0; i <= len(e)-len(tt.explanationContains); i++ { - if e[i:i+len(tt.explanationContains)] == tt.explanationContains { - return true - } - } + if strings.Contains(e, tt.explanationContains) { + return true } } return false From afbae2bef484f6ab35e964b093a882dc56c66c82 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 08:25:20 -0300 Subject: [PATCH 27/37] Remove outdated original spot check jobs plan --- docs/plans/spot-check-jobs.md | 439 ---------------------------------- 1 file changed, 439 deletions(-) delete mode 100644 docs/plans/spot-check-jobs.md diff --git a/docs/plans/spot-check-jobs.md b/docs/plans/spot-check-jobs.md deleted file mode 100644 index 2c467f8f0..000000000 --- a/docs/plans/spot-check-jobs.md +++ /dev/null @@ -1,439 +0,0 @@ -# Spot-Check Jobs in Component Readiness - -## Context - -Component Readiness is built around statistical analysis of **test pass rates** across -many runs of multiple prow job configurations. The sample is typically the last 7 days -and the basis is the 30 days prior to the last release's GA — a very stable period. -Fisher's Exact Test (or pass-rate thresholds) determines whether a regression is -statistically significant. - -We need to also monitor **rarely-run spot-check jobs** that verify obscure -configurations (vSphere hybrid installs, CPU partitioning, etcd scaling, etc.). These -jobs are run once and retried up to three times if they fail. The requirement is -simple: **the job must pass at least once in a 30-day window**. There is no basis -comparison, no Fisher's Exact, no confidence/minFail/pity thresholds — just a binary -"did it pass." - -These results must appear in the existing Component Readiness report grid, be -drillable via test details, and participate in regression tracking and triage. - -### Existing "Rarely Run" Concept - -There is an existing `VariantJunitTableOverride` mechanism -(`pkg/apis/config/v1/types.go:30-44`) that was intended for rarely-run jobs. It pulls -junit results from alternate BigQuery tables with extended date ranges. This is not -what we want — it is still based on test pass rates within rarely-run jobs, which is -too strict. We want **full job pass**, at least once, across a few retries. - -The existing `JobTier: rare` value will be replaced with `JobTier: spotcheck`. - -## Design - -### 1. New Job Variants - -Three changes to the variant system identify spot-check jobs: - -**New variant constants** in `pkg/variantregistry/ocp.go`: - -```go -const ( - VariantSpotCheckComponent = "SpotCheckComponent" - VariantSpotCheckCapability = "SpotCheckCapability" -) -``` - -**New setter function** `setSpotCheckComponent` registered in `IdentifyVariants` -**before** `setJobTier`. When matched, it atomically sets all three variants: - -```go -func setSpotCheckComponent(_ logrus.FieldLogger, variants map[string]string, jobName string) { - spotCheckPatterns := []struct { - substrings []string - component string - capability string - }{ - {[]string{"-cpu-partitioning"}, "Node", "CPU Partitioning"}, - {[]string{"-etcd-scaling"}, "etcd", "Scaling"}, - {[]string{"-vsphere-hybrid"}, "Installer", "vSphere Hybrid"}, - } - - for _, entry := range spotCheckPatterns { - if allSubstringsMatch(jobName, entry.substrings) { - variants[VariantSpotCheckComponent] = entry.component - variants[VariantSpotCheckCapability] = entry.capability - variants[VariantJobTier] = "spotcheck" - return - } - } -} -``` - -**Remove `"rare"` patterns** from `setJobTier` (lines 743-744). Jobs that were `rare` -either become `spotcheck` (via the new setter) or stay `candidate` until promoted. - -**Add to `importantVariants`** in `pkg/testidentification/ocp_variants.go` so they -flow through to the `job_variants` BigQuery table. - -**JobTier lifecycle:** - -``` -candidate → spotcheck - ↑ - add pattern to setSpotCheckComponent in ocp.go -``` - -### 2. View Configuration: Spot-Check Sample Window - -Views define a separate 30-day spot-check sample window. The normal 7-day test sample -window is unsuitable because spot-check jobs run infrequently. - -**New field in `pkg/apis/api/componentreport/crview/types.go`:** - -```go -type View struct { - // ... existing fields ... - SpotCheckSample *reqopts.RelativeRelease `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` -} -``` - -**Example in `config/views.yaml`:** - -```yaml -- name: 5.0-main - sample_release: - release: "5.0" - relative_start: now-7d - relative_end: now - spot_check_sample: - release: "5.0" - relative_start: now-30d - relative_end: now -``` - -**Carry resolved times through to `pkg/apis/api/componentreport/reqopts/types.go`:** - -```go -type RequestOptions struct { - // ... existing fields ... - SpotCheckSample *Release `json:"spot_check_sample,omitempty"` -} -``` - -**Resolve in `pkg/api/componentreadiness/utils/queryparamparser.go`:** -In `ParseComponentReportRequest`, after resolving view defaults (~line 50): - -```go -if view != nil && view.SpotCheckSample != nil { - resolved, err := GetViewReleaseOptions(releases, "spot_check", *view.SpotCheckSample, crTimeRoundingFactor) - if err != nil { - return - } - opts.SpotCheckSample = &resolved -} -``` - -Views **do not** add `"spotcheck"` to their `JobTier` include list. Normal test -queries filter on `JobTier: [blocking, informing, standard]`, which naturally excludes -spot-check jobs. The spot-check middleware independently queries with its own window. - -### 3. New Middleware: `spotcheckjobs` - -**Location:** `pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go` - -Implements `middleware.Middleware` (interface at -`pkg/api/componentreadiness/middleware/interface.go:15-40`). - -```go -type SpotCheckJobs struct { - dataProvider dataprovider.DataProvider - reqOptions reqopts.RequestOptions - log logrus.FieldLogger - - // Populated during QueryTestDetails, consumed by PreTestDetailsAnalysis - sampleJobDetails map[string][]dataprovider.JobRunDetail -} -``` - -#### Query Phase — Inject Synthetic Test Results - -Queries the BQ `jobs` table (joined with `job_variants`) for all jobs with -`JobTier = 'spotcheck'` in the spot-check sample window. Applies normal view variant -filters (Platform, Architecture, etc.) except JobTier. Groups results by -(SpotCheckComponent, SpotCheckCapability, column variant combo). - -Each group becomes one synthetic test result injected via `sampleStatusCh`: - -- **At least one pass:** `TestStatus{TotalCount: 1, SuccessCount: 1}` → green -- **No passes:** `TestStatus{TotalCount: 1, SuccessCount: 0}` → red - -No basis data is injected. The `Component` and `Capabilities` fields on `TestStatus` -come from the `SpotCheckComponent` / `SpotCheckCapability` variant values, placing the -synthetic test in the correct row of the CR grid. - -The synthetic test ID is deterministic: -`spotcheck::` - -The synthetic test name: -`[spot-check] / must pass at least once per sample window` - -#### PreAnalysis Phase — Bypass Fisher's, Set Status Directly - -```go -func (s *SpotCheckJobs) PreAnalysis(testKey crtest.Identification, - testStats *testdetails.TestComparison) error { - - if !strings.HasPrefix(testKey.TestID, "spotcheck:") { - return nil - } - - if testStats.SampleStats.SuccessCount > 0 { - testStats.ReportStatus = crtest.NotSignificant - testStats.Explanations = append(testStats.Explanations, - "Spot-check job passed at least once in the sample window") - } else { - testStats.ReportStatus = crtest.ExtremeRegression - testStats.Explanations = append(testStats.Explanations, - fmt.Sprintf("Spot-check job did not pass in the %d-day sample window", ...)) - } - - testStats.Comparison = crtest.SpotCheck - testStats.AnalysisComplete = true // signals assessComponentStatus to skip - return nil -} -``` - -This completely bypasses Fisher's Exact, confidence thresholds, minFail, pity factor, -and basis comparison. The middleware directly determines the outcome. - -#### QueryTestDetails Phase — Load Job Run Details - -Queries individual job runs (not aggregated) for spot-check jobs, storing them for -`PreTestDetailsAnalysis` to inject as synthetic `TestJobRunRows`. - -#### PreTestDetailsAnalysis Phase — Inject Job Runs for Drill-Down - -Injects the cached job run rows into `status.SampleStatus` so the test details page -renders individual job runs with pass/fail and prow links. - -#### PostAnalysis Phase — No-op - -The `RegressionTracker` middleware handles regression injection. The -`RegressionAllowances` middleware should skip spot-check tests (they have no basis). - -### 4. AnalysisComplete Flag - -**New field in `pkg/apis/api/componentreport/testdetails/types.go`:** - -```go -type TestComparison struct { - // ... existing fields ... - AnalysisComplete bool `json:"-"` -} -``` - -**Guard in `pkg/api/componentreadiness/component_report.go`:** - -```go -func (c *ComponentReportGenerator) assessComponentStatus( - testStats *testdetails.TestComparison, logger *log.Entry) { - if testStats.AnalysisComplete { - return - } - // ... existing Fisher's / pass-rate logic unchanged ... -} -``` - -This is a minimal change that allows any middleware to fully control analysis outcome. - -### 5. New Comparison Type - -**New constant in `pkg/apis/api/componentreport/crtest/types.go`:** - -```go -const ( - PassRate Comparison = "pass_rate" - FisherExact Comparison = "fisher_exact" - SpotCheck Comparison = "spot_check" // new -) -``` - -### 6. New BigQuery Queries - -**New methods on `DataProvider` interface -(`pkg/api/componentreadiness/dataprovider/interface.go`):** - -```go -type SpotCheckQuerier interface { - QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, - allJobVariants crtest.JobVariants, - start, end time.Time) ([]SpotCheckGroup, error) - - QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, - allJobVariants crtest.JobVariants, - spotCheckComponent, spotCheckCapability string, - variants map[string]string, - start, end time.Time) ([]JobRunDetail, error) -} - -type SpotCheckGroup struct { - SpotCheckComponent string - SpotCheckCapability string - Variants map[string]string - TotalRuns int - SuccessfulRuns int - JobNames []string -} - -type JobRunDetail struct { - JobName string - RunID string - URL string - StartTime time.Time - Success bool -} -``` - -**Component report query** (`QuerySpotCheckJobRuns`): - -```sql -SELECT - jv_SpotCheckComponent.variant_value AS spot_check_component, - jv_SpotCheckCapability.variant_value AS spot_check_capability, - -- dynamic column group by variants (Platform, Arch, Network, etc.) - COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, - COUNT(DISTINCT IF(jobs.prowjob_state = 'success', - jobs.prowjob_build_id, NULL)) AS successful_runs, - ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names -FROM {dataset}.jobs jobs - LEFT JOIN {dataset}.job_variants jv_SpotCheckComponent - ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name - AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent' - LEFT JOIN {dataset}.job_variants jv_SpotCheckCapability - ON jobs.prowjob_job_name = jv_SpotCheckCapability.job_name - AND jv_SpotCheckCapability.variant_name = 'SpotCheckCapability' - LEFT JOIN {dataset}.job_variants jv_Release ... - -- other variant joins for column group by -WHERE jobs.prowjob_start >= @From - AND jobs.prowjob_start < @To - AND jv_Release.variant_value = @Release - AND jv_SpotCheckComponent.variant_value IS NOT NULL - -- view include_variant filters (Platform, Architecture, etc.) -GROUP BY spot_check_component, spot_check_capability, variant_Platform, ... -``` - -Hits only the `jobs` table (joined with `job_variants`), never `junit`. Fast and cheap. - -**Test details query** (`QuerySpotCheckJobRunDetails`): Returns individual job runs -with pass/fail for a specific component/capability/variant combo. - -### 7. Middleware Registration - -In `pkg/api/componentreadiness/component_report.go`, `initializeMiddleware()`: - -```go -func (c *ComponentReportGenerator) initializeMiddleware() { - c.middlewares = middleware.List{} - - if c.ReqOptions.SpotCheckSample != nil { - c.middlewares = append(c.middlewares, - spotcheckjobs.NewSpotCheckJobsMiddleware(c.dataProvider, c.ReqOptions)) - } - - // ... existing middleware (releasefallback, regressiontracker, regressionallowances, linkinjector) ... -} -``` - -Spot-check middleware runs first so its synthetic results are in place before other -middleware processes them. - -### 8. Regression Tracking - -Spot-check regressions participate in the existing regression tracking pipeline -(`pkg/api/componentreadiness/regressiontracker.go:271-404`, -`SyncRegressionsForReport`). This works automatically because: - -- Synthetic spot-check tests appear in `report.Rows[].Columns[].RegressedTests` - like any other regressed test. -- `SyncRegressionsForReport` iterates all regressed tests and calls - `backend.OpenRegression(view, regTest)` for new ones. -- The synthetic test ID (`spotcheck:installer:vsphere-hybrid`) is stable and - deterministic, so regression records persist correctly across report runs. -- The `RegressionTracker` middleware (`PostAnalysis`) applies triage status - (triaged, fixed, failed-fixed) to spot-check regressions the same as normal ones. - -No changes needed to the regression tracking code. - -### 9. Frontend: `spot_check` Comparison Type - -**`sippy-ng/src/component_readiness/CompReadyTestDetailRow.js`:** - -The test details row currently renders pass_rate/successes/failures/flakes in the -`infoCell` function (lines 54-67). For `comparison === "spot_check"`, render a simpler -view: - -- Total job runs attempted -- Successful runs -- Required passes (1) -- No Fisher's Exact column -- No basis stats column - -**`sippy-ng/src/component_readiness/TestDetailsReport.js`:** - -When `data.analyses[0].comparison === "spot_check"`: -- Hide Fisher's Exact confidence display -- Hide basis release stats section -- Show explanation text prominently ("Spot-check job did not pass in 30-day window") -- Job run table still renders normally with prow links - -This is a small conditional — the page structure stays the same. - -## Files Changed - -| File | Change | -|------|--------| -| `pkg/variantregistry/ocp.go` | Add `setSpotCheckComponent` setter, new constants, remove `"rare"` patterns from `setJobTier`, register setter before `setJobTier` | -| `pkg/testidentification/ocp_variants.go` | Add `SpotCheckComponent`, `SpotCheckCapability` to `importantVariants` | -| `pkg/apis/api/componentreport/crview/types.go` | Add `SpotCheckSample *RelativeRelease` to `View` | -| `pkg/apis/api/componentreport/reqopts/types.go` | Add `SpotCheckSample *Release` to `RequestOptions` | -| `pkg/apis/api/componentreport/testdetails/types.go` | Add `AnalysisComplete bool` to `TestComparison` | -| `pkg/apis/api/componentreport/crtest/types.go` | Add `SpotCheck` comparison constant | -| `pkg/api/componentreadiness/dataprovider/interface.go` | Add `SpotCheckQuerier` interface, `SpotCheckGroup`, `JobRunDetail` types | -| `pkg/api/componentreadiness/dataprovider/bigquery/provider.go` | Implement `QuerySpotCheckJobRuns`, `QuerySpotCheckJobRunDetails` | -| `pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go` | **New file** — middleware implementation | -| `pkg/api/componentreadiness/component_report.go` | Add `AnalysisComplete` guard in `assessComponentStatus`, register middleware in `initializeMiddleware` | -| `pkg/api/componentreadiness/utils/queryparamparser.go` | Resolve `SpotCheckSample` from view config | -| `config/views.yaml` | Add `spot_check_sample` to relevant views | -| `sippy-ng/src/component_readiness/CompReadyTestDetailRow.js` | Conditional rendering for `spot_check` comparison | -| `sippy-ng/src/component_readiness/TestDetailsReport.js` | Hide Fisher/basis sections for `spot_check` | - -## What Doesn't Change - -- Fisher's Exact / pass-rate analysis logic (guarded by `AnalysisComplete`) -- Report types (`ComponentReport`, `ReportRow`, `ReportColumn`, `ReportTestSummary`) -- URL param structure (`testId`, `component`, `capability`, variant params all work) -- Caching strategy -- Regression tracking code (spot-check tests participate automatically) - -## Verification - -1. **Variant mapping:** After adding patterns to `setSpotCheckComponent`, run the - variant loader and verify jobs get `SpotCheckComponent`, `SpotCheckCapability`, - and `JobTier: spotcheck` variants in BigQuery's `job_variants` table. - -2. **Component report:** Load a view with `spot_check_sample` configured. Verify - spot-check tests appear in the correct component/capability rows. Break a - spot-check job (or test with a job that has no passes in the window) and verify - it shows as an extreme regression. - -3. **Test details drill-down:** Click a spot-check regression in the CR grid. Verify - the test details page shows individual job runs with prow links, no Fisher's - Exact stats, and the spot-check explanation text. - -4. **Regression tracking:** Verify a failing spot-check job creates a - `test_regressions` record. Verify triaging the regression via the normal triage - flow works (status changes to triaged/fixed/failed-fixed). - -5. **No interference:** Verify normal test-based regressions are unaffected — - Fisher's Exact still runs, basis comparison still works, existing views show the - same results. From b7afda45ecd044920de39b261e37ea2c9e1b073d Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 08:30:01 -0300 Subject: [PATCH 28/37] Remove spot-check fallback SQL and legacy 'rare' tier support The transition period is over: all spot-check jobs now carry SpotCheckComponent and SpotCheckCapability variants in the job_variants table. Remove the COALESCE job-name-based fallbacks and drop 'rare' from the JobTier filter in both spot-check queries. Co-Authored-By: Claude Sonnet 4.6 --- .../dataprovider/bigquery/provider.go | 38 ++++--------------- 1 file changed, 8 insertions(+), 30 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index ca683d47f..87cb248c9 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -377,27 +377,10 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // --- SpotCheckQuerier --- // spotCheckFallbackSQL provides a CASE expression for deriving component/capability -// from job names when the SpotCheckComponent/SpotCheckCapability variants are not yet -// populated in the job_variants table. This fallback can be removed once the variant -// syncer has run and all spot-check jobs have the new variants. -const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, - CASE - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node / Kubelet' - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Etcd' - END)` - -const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, - CASE - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'CPU Partitioning' - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Scaling' - END)` - // QuerySpotCheckJobRuns queries the jobs table (not junit) for spotcheck-tier periodic/release // jobs, grouping by component, capability, and the requested variant columns. // Returns aggregated pass/fail counts per group so the middleware can create // synthetic test results without needing individual test case data. -// During the transition period, also matches jobs with the legacy 'rare' tier and uses -// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { @@ -466,8 +449,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions queryString := fmt.Sprintf(` SELECT %s - %s AS spot_check_component, - %s AS spot_check_capability, + jv_SpotCheckComponent.variant_value AS spot_check_component, + jv_SpotCheckCapability.variant_value AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names, @@ -477,13 +460,12 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value IN ('spotcheck', 'rare') + AND jv_JobTier.variant_value = 'spotcheck' AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s GROUP BY %s, spot_check_component, spot_check_capability HAVING spot_check_component IS NOT NULL AND spot_check_capability IS NOT NULL - `, selectVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, - p.client.Dataset, joinVariants, variantFilters, + `, selectVariants, p.client.Dataset, joinVariants, variantFilters, groupByVariants) params = append(params, @@ -550,8 +532,6 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions // QuerySpotCheckJobRunDetails returns individual job runs matching the given variant // and component/capability filters, used to populate the test details drill-down page // for a specific spot-check synthetic test. -// During the transition period, also matches jobs with the legacy 'rare' tier and uses -// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, @@ -598,15 +578,13 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value IN ('spotcheck', 'rare') + AND jv_JobTier.variant_value = 'spotcheck' AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') - AND LOWER(%s) = LOWER(@SpotCheckComponent) - AND LOWER(%s) = LOWER(@SpotCheckCapability) + AND LOWER(jv_SpotCheckComponent.variant_value) = LOWER(@SpotCheckComponent) + AND LOWER(jv_SpotCheckCapability.variant_value) = LOWER(@SpotCheckCapability) %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, - spotCheckComponentFallback, spotCheckCapabilityFallback, - variantFilters) + `, p.client.Dataset, joinVariants, variantFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, From 835678af7930f874febacedfd4ccc6cd25cf40b4 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 12:21:05 -0300 Subject: [PATCH 29/37] Use dbgroupby for same variants as normal regressions --- .../dataprovider/bigquery/provider.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 87cb248c9..eb2e09a1c 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -385,14 +385,14 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions allJobVariants crtest.JobVariants, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { - columnGroupByVariants := reqOptions.VariantOption.ColumnGroupBy - if len(columnGroupByVariants) == 0 { - columnGroupByVariants = sets.NewString("Platform", "Architecture", "Network") + groupByVariantSet := reqOptions.VariantOption.DBGroupBy + if len(groupByVariantSet) == 0 { + groupByVariantSet = sets.NewString("Platform", "Architecture", "Network") } var selectVariants, joinVariants string var groupByParts []string - for _, v := range columnGroupByVariants.List() { + for _, v := range groupByVariantSet.List() { cleanV := param.Cleanse(v) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", @@ -417,7 +417,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions p.client.Dataset) // Track which variant groups already have JOINs - joinedVariants := sets.NewString(columnGroupByVariants.List()...) + joinedVariants := sets.NewString(groupByVariantSet.List()...) joinedVariants.Insert("Release", "JobTier", "SpotCheckComponent", "SpotCheckCapability") // Build include variant filters (Platform, Architecture, etc.) but skip JobTier and SpotCheck variants @@ -502,7 +502,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions if val, ok := rawRow["spot_check_capability"]; ok && val != nil { group.Capability = val.(string) } - for _, v := range columnGroupByVariants.List() { + for _, v := range groupByVariantSet.List() { cleanV := param.Cleanse(v) if val, ok := rawRow["variant_"+cleanV]; ok && val != nil { group.Variants[v] = val.(string) From a24ce2813d42034ac6f4c3332bf5aa18ec8f7d7e Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 12:21:15 -0300 Subject: [PATCH 30/37] Revert "Remove spot-check fallback SQL and legacy 'rare' tier support" This reverts commit b7afda45ecd044920de39b261e37ea2c9e1b073d. --- .../dataprovider/bigquery/provider.go | 38 +++++++++++++++---- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index eb2e09a1c..6ee56ee93 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -377,10 +377,27 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // --- SpotCheckQuerier --- // spotCheckFallbackSQL provides a CASE expression for deriving component/capability +// from job names when the SpotCheckComponent/SpotCheckCapability variants are not yet +// populated in the job_variants table. This fallback can be removed once the variant +// syncer has run and all spot-check jobs have the new variants. +const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node / Kubelet' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Etcd' + END)` + +const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'CPU Partitioning' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Scaling' + END)` + // QuerySpotCheckJobRuns queries the jobs table (not junit) for spotcheck-tier periodic/release // jobs, grouping by component, capability, and the requested variant columns. // Returns aggregated pass/fail counts per group so the middleware can create // synthetic test results without needing individual test case data. +// During the transition period, also matches jobs with the legacy 'rare' tier and uses +// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { @@ -449,8 +466,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions queryString := fmt.Sprintf(` SELECT %s - jv_SpotCheckComponent.variant_value AS spot_check_component, - jv_SpotCheckCapability.variant_value AS spot_check_capability, + %s AS spot_check_component, + %s AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names, @@ -460,12 +477,13 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value = 'spotcheck' + AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s GROUP BY %s, spot_check_component, spot_check_capability HAVING spot_check_component IS NOT NULL AND spot_check_capability IS NOT NULL - `, selectVariants, p.client.Dataset, joinVariants, variantFilters, + `, selectVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, + p.client.Dataset, joinVariants, variantFilters, groupByVariants) params = append(params, @@ -532,6 +550,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions // QuerySpotCheckJobRunDetails returns individual job runs matching the given variant // and component/capability filters, used to populate the test details drill-down page // for a specific spot-check synthetic test. +// During the transition period, also matches jobs with the legacy 'rare' tier and uses +// COALESCE fallback SQL to derive component/capability from job name substrings. func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, variants map[string]string, @@ -578,13 +598,15 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value = 'spotcheck' + AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') - AND LOWER(jv_SpotCheckComponent.variant_value) = LOWER(@SpotCheckComponent) - AND LOWER(jv_SpotCheckCapability.variant_value) = LOWER(@SpotCheckCapability) + AND LOWER(%s) = LOWER(@SpotCheckComponent) + AND LOWER(%s) = LOWER(@SpotCheckCapability) %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, variantFilters) + `, p.client.Dataset, joinVariants, + spotCheckComponentFallback, spotCheckCapabilityFallback, + variantFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, From 97351f471ad2da91d3cb16b22812e0af0fffc03d Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 12 Jun 2026 14:24:42 -0300 Subject: [PATCH 31/37] Support potentially multiple spot_check_sample ranges in future --- config/views.yaml | 12 +- .../componentreadiness/component_report.go | 2 +- .../dataprovider/bigquery/provider.go | 113 ++++++++----- .../dataprovider/interface.go | 9 +- .../dataprovider/postgres/provider.go | 4 +- .../middleware/spotcheckjobs/spotcheckjobs.go | 160 ++++++++++-------- .../spotcheckjobs/spotcheckjobs_test.go | 41 +++-- pkg/api/componentreadiness/test_details.go | 2 +- .../utils/queryparamparser.go | 34 ++-- pkg/apis/api/componentreport/crview/types.go | 18 +- pkg/apis/api/componentreport/reqopts/types.go | 14 +- .../regressioncacheloader.go | 14 +- pkg/variantregistry/ocp.go | 8 +- pkg/variantregistry/ocp_test.go | 8 +- pkg/variantregistry/snapshot.yaml | 144 ++++++++-------- 15 files changed, 332 insertions(+), 251 deletions(-) diff --git a/config/views.yaml b/config/views.yaml index d5280becf..8a4b450e2 100755 --- a/config/views.yaml +++ b/config/views.yaml @@ -9,9 +9,13 @@ component_readiness: release: "5.0" relative_start: now-7d relative_end: now - spot_check_sample: - relative_start: now-30d - relative_end: now + spot_check_job_samples: + spotcheck-30d: + relative_start: now-30d + relative_end: now + include_variants: + JobTier: + - spotcheck-30d variant_options: column_group_by: Network: { } @@ -43,7 +47,7 @@ component_readiness: - blocking - informing - standard - - spotcheck + - spotcheck-30d LayeredProduct: - none - virt diff --git a/pkg/api/componentreadiness/component_report.go b/pkg/api/componentreadiness/component_report.go index 47dfeefc4..8aa6c8fc0 100644 --- a/pkg/api/componentreadiness/component_report.go +++ b/pkg/api/componentreadiness/component_report.go @@ -280,7 +280,7 @@ func (c *ComponentReportGenerator) initializeMiddleware() { // Initialize all our middleware applicable to this request. // middlewares that inject synthetic tests must run first so results are in place for other middleware. - if c.ReqOptions.SpotCheckSample != nil { + if len(c.ReqOptions.SpotCheckJobSamples) > 0 { c.middlewares = append(c.middlewares, spotcheckjobs.NewSpotCheckJobsMiddleware(c.dataProvider, c.ReqOptions)) } diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 6ee56ee93..12c268fbe 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -380,26 +380,15 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // from job names when the SpotCheckComponent/SpotCheckCapability variants are not yet // populated in the job_variants table. This fallback can be removed once the variant // syncer has run and all spot-check jobs have the new variants. -const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, - CASE - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node / Kubelet' - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Etcd' - END)` - -const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, - CASE - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'CPU Partitioning' - WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Scaling' - END)` - -// QuerySpotCheckJobRuns queries the jobs table (not junit) for spotcheck-tier periodic/release -// jobs, grouping by component, capability, and the requested variant columns. +// QuerySpotCheckJobRuns queries the jobs table (not junit) for spot-check periodic/release +// jobs, grouping by component, capability, and the DB group-by variants. // Returns aggregated pass/fail counts per group so the middleware can create // synthetic test results without needing individual test case data. -// During the transition period, also matches jobs with the legacy 'rare' tier and uses -// COALESCE fallback SQL to derive component/capability from job name substrings. +// includeVariants specifies variant filters (ANDed) from the spot-check sample config, +// typically including JobTier to select the correct spot-check tier. func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + spotCheckIncludeVariants map[string][]string, start, end time.Time) ([]dataprovider.SpotCheckGroup, error) { groupByVariantSet := reqOptions.VariantOption.DBGroupBy @@ -419,13 +408,10 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions } groupByVariants := strings.Join(groupByParts, ", ") - // Always join Release, JobTier, SpotCheckComponent, SpotCheckCapability + // Always join Release, SpotCheckComponent, SpotCheckCapability joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_Release ON jobs.prowjob_job_name = jv_Release.job_name AND jv_Release.variant_name = 'Release'\n", p.client.Dataset) - joinVariants += fmt.Sprintf( - "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", - p.client.Dataset) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_SpotCheckComponent ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent'\n", p.client.Dataset) @@ -435,16 +421,18 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions // Track which variant groups already have JOINs joinedVariants := sets.NewString(groupByVariantSet.List()...) - joinedVariants.Insert("Release", "JobTier", "SpotCheckComponent", "SpotCheckCapability") + joinedVariants.Insert("Release", "SpotCheckComponent", "SpotCheckCapability") - // Build include variant filters (Platform, Architecture, etc.) but skip JobTier and SpotCheck variants + // Build variant filters from both view-level and spot-check sample-level include_variants variantFilters := "" var params []bigquery.QueryParameter - includeVariants := reqOptions.VariantOption.IncludeVariants - if includeVariants == nil { - includeVariants = map[string][]string{} + + // Apply view-level include variant filters, but skip SpotCheck-specific variants + viewIncludeVariants := reqOptions.VariantOption.IncludeVariants + if viewIncludeVariants == nil { + viewIncludeVariants = map[string][]string{} } - for _, group := range sortedKeys(includeVariants) { + for _, group := range sortedKeys(viewIncludeVariants) { if group == "JobTier" || group == "SpotCheckComponent" || group == "SpotCheckCapability" { continue } @@ -459,15 +447,32 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) params = append(params, bigquery.QueryParameter{ Name: paramName, - Value: includeVariants[group], + Value: viewIncludeVariants[group], + }) + } + + // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d) + for _, group := range sortedKeys(spotCheckIncludeVariants) { + cleanGroup := param.Cleanse(group) + if !joinedVariants.Has(group) { + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + joinedVariants.Insert(group) + } + paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) + variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: spotCheckIncludeVariants[group], }) } queryString := fmt.Sprintf(` SELECT %s - %s AS spot_check_component, - %s AS spot_check_capability, + jv_SpotCheckComponent.variant_value AS spot_check_component, + jv_SpotCheckCapability.variant_value AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names, @@ -477,13 +482,11 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') %s GROUP BY %s, spot_check_component, spot_check_capability HAVING spot_check_component IS NOT NULL AND spot_check_capability IS NOT NULL - `, selectVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, - p.client.Dataset, joinVariants, variantFilters, + `, selectVariants, p.client.Dataset, joinVariants, variantFilters, groupByVariants) params = append(params, @@ -550,10 +553,10 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions // QuerySpotCheckJobRunDetails returns individual job runs matching the given variant // and component/capability filters, used to populate the test details drill-down page // for a specific spot-check synthetic test. -// During the transition period, also matches jobs with the legacy 'rare' tier and uses -// COALESCE fallback SQL to derive component/capability from job name substrings. +// includeVariants specifies variant filters (ANDed) from the spot-check sample config. func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + spotCheckIncludeVariants map[string][]string, variants map[string]string, component, capability string, start, end time.Time) ([]dataprovider.JobRunDetail, error) { @@ -561,9 +564,6 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO joinVariants := fmt.Sprintf( "LEFT JOIN %s.job_variants jv_Release ON jobs.prowjob_job_name = jv_Release.job_name AND jv_Release.variant_name = 'Release'\n", p.client.Dataset) - joinVariants += fmt.Sprintf( - "LEFT JOIN %s.job_variants jv_JobTier ON jobs.prowjob_job_name = jv_JobTier.job_name AND jv_JobTier.variant_name = 'JobTier'\n", - p.client.Dataset) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_SpotCheckComponent ON jobs.prowjob_job_name = jv_SpotCheckComponent.job_name AND jv_SpotCheckComponent.variant_name = 'SpotCheckComponent'\n", p.client.Dataset) @@ -571,13 +571,20 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO "LEFT JOIN %s.job_variants jv_SpotCheckCapability ON jobs.prowjob_job_name = jv_SpotCheckCapability.job_name AND jv_SpotCheckCapability.variant_name = 'SpotCheckCapability'\n", p.client.Dataset) + joinedVariants := sets.NewString("Release", "SpotCheckComponent", "SpotCheckCapability") + variantFilters := "" var params []bigquery.QueryParameter + + // Join and filter by the requested variant dimensions (e.g. Platform, Architecture) for k, v := range variants { cleanK := param.Cleanse(k) - joinVariants += fmt.Sprintf( - "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanK, cleanK, cleanK, k) + if !joinedVariants.Has(k) { + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanK, cleanK, cleanK, k) + joinedVariants.Insert(k) + } paramName := fmt.Sprintf("variant_%s", cleanK) variantFilters += fmt.Sprintf(" AND jv_%s.variant_value = @%s", cleanK, paramName) params = append(params, bigquery.QueryParameter{ @@ -586,6 +593,23 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO }) } + // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d) + for _, group := range sortedKeys(spotCheckIncludeVariants) { + cleanGroup := param.Cleanse(group) + if !joinedVariants.Has(group) { + joinVariants += fmt.Sprintf( + "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + joinedVariants.Insert(group) + } + paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) + variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) + params = append(params, bigquery.QueryParameter{ + Name: paramName, + Value: spotCheckIncludeVariants[group], + }) + } + queryString := fmt.Sprintf(` SELECT jobs.prowjob_job_name AS job_name, @@ -598,15 +622,12 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO WHERE jobs.prowjob_start >= DATETIME(@From) AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release - AND jv_JobTier.variant_value IN ('spotcheck', 'rare') AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') - AND LOWER(%s) = LOWER(@SpotCheckComponent) - AND LOWER(%s) = LOWER(@SpotCheckCapability) + AND LOWER(jv_SpotCheckComponent.variant_value) = LOWER(@SpotCheckComponent) + AND LOWER(jv_SpotCheckCapability.variant_value) = LOWER(@SpotCheckCapability) %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, - spotCheckComponentFallback, spotCheckCapabilityFallback, - variantFilters) + `, p.client.Dataset, joinVariants, variantFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, diff --git a/pkg/api/componentreadiness/dataprovider/interface.go b/pkg/api/componentreadiness/dataprovider/interface.go index e4757442a..78081d867 100644 --- a/pkg/api/componentreadiness/dataprovider/interface.go +++ b/pkg/api/componentreadiness/dataprovider/interface.go @@ -69,18 +69,19 @@ type JobQuerier interface { // SpotCheckQuerier fetches job-level pass/fail data for spot-check analysis. type SpotCheckQuerier interface { // QuerySpotCheckJobRuns returns aggregated pass/fail per spot-check group, - // grouped by SpotCheckComponent, SpotCheckCapability, and the column group-by variants. - // Queries the jobs table, not junit. During the transition period, falls back to - // job name substring matching when SpotCheckComponent/SpotCheckCapability variants - // are not yet populated in the job_variants table. + // grouped by SpotCheckComponent, SpotCheckCapability, and the DB group-by variants. + // includeVariants specifies variant filters (ANDed) from the spot-check sample config. QuerySpotCheckJobRuns(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + includeVariants map[string][]string, start, end time.Time) ([]SpotCheckGroup, error) // QuerySpotCheckJobRunDetails returns individual job runs for a specific // spot-check group, used for test details drill-down. + // includeVariants specifies variant filters (ANDed) from the spot-check sample config. QuerySpotCheckJobRunDetails(ctx context.Context, reqOptions reqopts.RequestOptions, allJobVariants crtest.JobVariants, + includeVariants map[string][]string, variants map[string]string, component, capability string, start, end time.Time) ([]JobRunDetail, error) diff --git a/pkg/api/componentreadiness/dataprovider/postgres/provider.go b/pkg/api/componentreadiness/dataprovider/postgres/provider.go index a7bde3cbb..be1b6fbcc 100644 --- a/pkg/api/componentreadiness/dataprovider/postgres/provider.go +++ b/pkg/api/componentreadiness/dataprovider/postgres/provider.go @@ -801,11 +801,11 @@ func (p *PostgresProvider) LookupJobVariants(ctx context.Context, jobName string // Postgres does not support spot-check queries; these are BigQuery-only. func (p *PostgresProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + _ crtest.JobVariants, _ map[string][]string, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { return nil, nil } func (p *PostgresProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + _ crtest.JobVariants, _ map[string][]string, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { return nil, nil } diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go index c9c195441..36962b162 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs.go @@ -25,6 +25,10 @@ var _ middleware.Middleware = &SpotCheckJobs{} // Jobs are identified by the SpotCheckComponent and SpotCheckCapability variants in the // variant registry. The component/capability values control where the synthetic test // results appear in the component readiness report. +// +// Multiple spot-check samples can be configured (e.g. spotcheck-30d, spotcheck-90d), +// each with its own time window and variant filters. Each sample produces a separate +// set of synthetic test results. func NewSpotCheckJobsMiddleware( provider dataprovider.DataProvider, reqOptions reqopts.RequestOptions, @@ -46,71 +50,72 @@ type SpotCheckJobs struct { sampleJobDetailsMutex sync.Mutex } -// Query fetches aggregated spot-check job results from BigQuery, creates synthetic -// test statuses (one per component/capability/variant group), and injects them into -// the sample status channel. Each synthetic test uses a binary pass/fail: -// >=1 successful run = pass. +// Query fetches aggregated spot-check job results from BigQuery for each configured +// sample, creates synthetic test statuses (one per component/capability/variant group +// per sample), and injects them into the sample status channel. func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, allJobVariants crtest.JobVariants, _, sampleStatusCh chan map[string]crstatus.TestStatus, errCh chan error) { - if s.reqOptions.SpotCheckSample == nil { + if len(s.reqOptions.SpotCheckJobSamples) == 0 { return } - wg.Go(func() { - select { - case <-ctx.Done(): - s.log.Info("context canceled during spot check query") - return - default: - } - - groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, - s.reqOptions, allJobVariants, - s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) - if err != nil { - errCh <- fmt.Errorf("spot check query failed: %w", err) - return - } - - sampleStatus := map[string]crstatus.TestStatus{} - requestedVariants := map[string]string{} - if len(s.reqOptions.TestIDOptions) > 0 { - requestedVariants = s.reqOptions.TestIDOptions[0].RequestedVariants - } - for _, group := range groups { - if group.Component == "" || group.Capability == "" { - s.log.Warnf("skipping spot-check group with empty component/capability: %+v", group) - continue + for _, sample := range s.reqOptions.SpotCheckJobSamples { + wg.Go(func() { + select { + case <-ctx.Done(): + s.log.Info("context canceled during spot check query") + return + default: } - if !variantsMatch(group.Variants, requestedVariants) { - continue + groups, err := s.dataProvider.QuerySpotCheckJobRuns(ctx, + s.reqOptions, allJobVariants, sample.IncludeVariants, + sample.Start, sample.End) + if err != nil { + errCh <- fmt.Errorf("spot check query failed for %s: %w", sample.Name, err) + return } - testKey := crtest.KeyWithVariants{ - TestID: syntheticTestID(group.Component, group.Capability), - Variants: group.Variants, + sampleStatus := map[string]crstatus.TestStatus{} + requestedVariants := map[string]string{} + if len(s.reqOptions.TestIDOptions) > 0 { + requestedVariants = s.reqOptions.TestIDOptions[0].RequestedVariants } - keyStr := testKey.KeyOrDie() - - sampleStatus[keyStr] = crstatus.TestStatus{ - TestName: syntheticTestName(group.Component, group.Capability), - Component: group.Component, - Capabilities: []string{group.Capability}, - Variants: variantMapToSlice(group.Variants), - Count: crtest.Count{ - TotalCount: group.TotalRuns, - SuccessCount: group.SuccessfulRuns, - }, - LastFailure: group.LastFailure, + for _, group := range groups { + if group.Component == "" || group.Capability == "" { + s.log.Warnf("skipping spot-check group with empty component/capability: %+v", group) + continue + } + + if !variantsMatch(group.Variants, requestedVariants) { + continue + } + + testKey := crtest.KeyWithVariants{ + TestID: syntheticTestID(sample.Name, group.Component, group.Capability), + Variants: group.Variants, + } + keyStr := testKey.KeyOrDie() + + sampleStatus[keyStr] = crstatus.TestStatus{ + TestName: syntheticTestName(group.Component, group.Capability), + Component: group.Component, + Capabilities: []string{group.Capability}, + Variants: variantMapToSlice(group.Variants), + Count: crtest.Count{ + TotalCount: group.TotalRuns, + SuccessCount: group.SuccessfulRuns, + }, + LastFailure: group.LastFailure, + } } - } - s.log.Infof("injecting %d spot-check synthetic test results", len(sampleStatus)) - sampleStatusCh <- sampleStatus - }) + s.log.Infof("injecting %d spot-check synthetic test results for sample %s", len(sampleStatus), sample.Name) + sampleStatusCh <- sampleStatus + }) + } } // QueryTestDetails fetches individual job run details for spot-check synthetic tests, @@ -118,7 +123,7 @@ func (s *SpotCheckJobs) Query(ctx context.Context, wg *sync.WaitGroup, func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup, errCh chan error, allJobVariants crtest.JobVariants) { - if s.reqOptions.SpotCheckSample == nil { + if len(s.reqOptions.SpotCheckJobSamples) == 0 { return } @@ -127,12 +132,18 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup continue } - component, capability := componentCapabilityFromTestID(testIDOpt.TestID) + sampleName, component, capability := componentCapabilityFromTestID(testIDOpt.TestID) if component == "" || capability == "" { s.log.Warnf("could not parse component/capability from spot-check test ID %s", testIDOpt.TestID) continue } + sample := s.findSample(sampleName) + if sample == nil { + s.log.Warnf("no spot-check sample found for tier %s in test ID %s", sampleName, testIDOpt.TestID) + continue + } + wg.Go(func() { select { case <-ctx.Done(): @@ -141,9 +152,9 @@ func (s *SpotCheckJobs) QueryTestDetails(ctx context.Context, wg *sync.WaitGroup } details, err := s.dataProvider.QuerySpotCheckJobRunDetails(ctx, - s.reqOptions, allJobVariants, + s.reqOptions, allJobVariants, sample.IncludeVariants, testIDOpt.RequestedVariants, component, capability, - s.reqOptions.SpotCheckSample.Start, s.reqOptions.SpotCheckSample.End) + sample.Start, sample.End) if err != nil { errCh <- fmt.Errorf("spot check details query failed: %w", err) return @@ -185,7 +196,13 @@ func (s *SpotCheckJobs) Analyze(testKey crtest.Identification, return false, nil } - sampleDays := int(s.reqOptions.SpotCheckSample.End.Sub(s.reqOptions.SpotCheckSample.Start).Hours() / 24) + sampleName, _, _ := componentCapabilityFromTestID(testKey.TestID) + sample := s.findSample(sampleName) + sampleDays := 0 + if sample != nil { + sampleDays = int(sample.End.Sub(sample.Start).Hours() / 24) + } + totalRuns := testStats.SampleStats.Total() successfulRuns := testStats.SampleStats.SuccessCount failedRuns := totalRuns - successfulRuns @@ -270,29 +287,34 @@ func (s *SpotCheckJobs) PreTestDetailsAnalysis(testKey crtest.KeyWithVariants, return nil } +func (s *SpotCheckJobs) findSample(name string) *reqopts.SpotCheckJobSampleOpts { + for i := range s.reqOptions.SpotCheckJobSamples { + if s.reqOptions.SpotCheckJobSamples[i].Name == name { + return &s.reqOptions.SpotCheckJobSamples[i] + } + } + return nil +} + func isSpotCheckTestID(testID string) bool { - return strings.HasPrefix(testID, "spotcheck:") + return strings.HasPrefix(testID, "spotcheck-") } -// componentCapabilityFromTestID extracts the component and capability from a synthetic -// spot-check test ID. The format is "spotcheck::" where the -// component is lowercased and capability has spaces replaced with dashes. -func componentCapabilityFromTestID(testID string) (string, string) { +// componentCapabilityFromTestID extracts the sample name, component and capability from +// a synthetic spot-check test ID. The format is "spotcheck-30d:component:capability". +func componentCapabilityFromTestID(testID string) (string, string, string) { parts := strings.SplitN(testID, ":", 3) if len(parts) != 3 { - return "", "" + return "", "", "" } - // The test ID stores lowercased component and dash-separated capability. - // We need to convert back to the original format for the BigQuery query. - // The COALESCE fallback in the query matches exact component/capability values, - // so we need title case for component and space-separated for capability. component := parts[1] capability := strings.ReplaceAll(parts[2], "-", " ") - return component, capability + return parts[0], component, capability } -func syntheticTestID(component, capability string) string { - return fmt.Sprintf("spotcheck:%s:%s", +func syntheticTestID(sampleName, component, capability string) string { + return fmt.Sprintf("%s:%s:%s", + sampleName, strings.ToLower(component), strings.ToLower(strings.ReplaceAll(capability, " ", "-"))) } diff --git a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go index 6f49cdc91..91d53b846 100644 --- a/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go +++ b/pkg/api/componentreadiness/middleware/spotcheckjobs/spotcheckjobs_test.go @@ -25,12 +25,12 @@ type stubProvider struct { } func (s *stubProvider) QuerySpotCheckJobRuns(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { + _ crtest.JobVariants, _ map[string][]string, _, _ time.Time) ([]dataprovider.SpotCheckGroup, error) { return s.groups, nil } func (s *stubProvider) QuerySpotCheckJobRunDetails(_ context.Context, _ reqopts.RequestOptions, - _ crtest.JobVariants, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { + _ crtest.JobVariants, _ map[string][]string, _ map[string]string, _, _ string, _, _ time.Time) ([]dataprovider.JobRunDetail, error) { return nil, nil } @@ -111,9 +111,20 @@ func TestVariantsMatch(t *testing.T) { } } -func TestQueryFiltering(t *testing.T) { - spotCheckSample := reqopts.Release{Start: time.Now().Add(-30 * 24 * time.Hour), End: time.Now()} +func newSpotCheckSamples() []reqopts.SpotCheckJobSampleOpts { + return []reqopts.SpotCheckJobSampleOpts{ + { + Name: "spotcheck-30d", + Release: reqopts.Release{ + Start: time.Now().Add(-30 * 24 * time.Hour), + End: time.Now(), + }, + IncludeVariants: map[string][]string{"JobTier": {"spotcheck-30d"}}, + }, + } +} +func TestQueryFiltering(t *testing.T) { allGroups := []dataprovider.SpotCheckGroup{ {Component: "Etcd", Capability: "Scaling", Variants: map[string]string{"Network": "ovn", "Platform": "aws", "Topology": "ha"}, TotalRuns: 3, SuccessfulRuns: 2}, {Component: "Etcd", Capability: "Scaling", Variants: map[string]string{"Network": "ovn", "Platform": "gcp", "Topology": "ha"}, TotalRuns: 3, SuccessfulRuns: 3}, @@ -130,18 +141,18 @@ func TestQueryFiltering(t *testing.T) { { name: "no filter - all groups injected", requestedVariants: map[string]string{}, - expectedKeys: []string{"spotcheck:etcd:scaling", "spotcheck:node / kubelet:cpu-partitioning"}, + expectedKeys: []string{"spotcheck-30d:etcd:scaling", "spotcheck-30d:node / kubelet:cpu-partitioning"}, }, { name: "environment filter Platform=aws - excludes gcp", requestedVariants: map[string]string{"Platform": "aws"}, - expectedKeys: []string{"spotcheck:etcd:scaling", "spotcheck:node / kubelet:cpu-partitioning"}, + expectedKeys: []string{"spotcheck-30d:etcd:scaling", "spotcheck-30d:node / kubelet:cpu-partitioning"}, notExpectedKeys: []string{}, }, { name: "environment filter Platform=gcp - only gcp etcd", requestedVariants: map[string]string{"Platform": "gcp"}, - notExpectedKeys: []string{"spotcheck:node / kubelet:cpu-partitioning"}, + notExpectedKeys: []string{"spotcheck-30d:node / kubelet:cpu-partitioning"}, }, { name: "environment filter Platform=metal - nothing matches", @@ -155,7 +166,7 @@ func TestQueryFiltering(t *testing.T) { mw := &SpotCheckJobs{ dataProvider: &stubProvider{groups: allGroups}, reqOptions: reqopts.RequestOptions{ - SpotCheckSample: &spotCheckSample, + SpotCheckJobSamples: newSpotCheckSamples(), TestIDOptions: []reqopts.TestIdentification{ {Component: tt.requestedComp, RequestedVariants: tt.requestedVariants}, }, @@ -189,20 +200,20 @@ func TestAnalyze(t *testing.T) { sampleStart := now.Add(-30 * 24 * time.Hour) sampleEnd := now - spotCheckSample := reqopts.Release{ - Start: sampleStart, - End: sampleEnd, - } - mw := &SpotCheckJobs{ reqOptions: reqopts.RequestOptions{ - SpotCheckSample: &spotCheckSample, + SpotCheckJobSamples: []reqopts.SpotCheckJobSampleOpts{ + { + Name: "spotcheck-30d", + Release: reqopts.Release{Start: sampleStart, End: sampleEnd}, + }, + }, }, } spotCheckTestKey := crtest.Identification{ RowIdentification: crtest.RowIdentification{ - TestID: "spotcheck:etcd:scaling", + TestID: "spotcheck-30d:etcd:scaling", }, } diff --git a/pkg/api/componentreadiness/test_details.go b/pkg/api/componentreadiness/test_details.go index 18b5f9605..c2b3bd69c 100644 --- a/pkg/api/componentreadiness/test_details.go +++ b/pkg/api/componentreadiness/test_details.go @@ -195,7 +195,7 @@ func (c *ComponentReportGenerator) GenerateDetailsReportForTest( if testIDOption.TestID == "" { return testdetails.Report{}, []error{fmt.Errorf("test_id has to be defined for test details")} } - isSpotCheck := strings.HasPrefix(testIDOption.TestID, "spotcheck:") + isSpotCheck := strings.HasPrefix(testIDOption.TestID, "spotcheck-") if !isSpotCheck { for _, v := range c.ReqOptions.VariantOption.DBGroupBy.List() { if _, ok := testIDOption.RequestedVariants[v]; !ok { diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index 61acbbaf1..7882869b2 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -56,18 +56,22 @@ func ParseComponentReportRequest( warnings = append(warnings, viewWarnings...) } - if view.SpotCheckSample != nil { + for name, sample := range view.SpotCheckJobSamples { spotCheckRelative := reqopts.RelativeRelease{ Release: reqopts.Release{Name: view.SampleRelease.Name}, - RelativeStart: view.SpotCheckSample.RelativeStart, - RelativeEnd: view.SpotCheckSample.RelativeEnd, + RelativeStart: sample.RelativeStart, + RelativeEnd: sample.RelativeEnd, } - resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check", spotCheckRelative, crTimeRoundingFactor, crTimeRoundingOffset) + resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check_"+name, spotCheckRelative, crTimeRoundingFactor, crTimeRoundingOffset) if resolveErr != nil { err = resolveErr return } - opts.SpotCheckSample = &resolved + opts.SpotCheckJobSamples = append(opts.SpotCheckJobSamples, reqopts.SpotCheckJobSampleOpts{ + Name: name, + Release: resolved, + IncludeVariants: sample.IncludeVariants, + }) } } @@ -224,13 +228,21 @@ func ParseComponentReportRequest( } } - // If no view provided SpotCheckSample, default it from the sample release dates + // If no view provided SpotCheckJobSamples, default from the sample release dates // so spot-check middleware runs on drill-down requests too. - if opts.SpotCheckSample == nil && !opts.SampleRelease.Start.IsZero() && !opts.SampleRelease.End.IsZero() { - opts.SpotCheckSample = &reqopts.Release{ - Name: opts.SampleRelease.Name, - Start: opts.SampleRelease.Start, - End: opts.SampleRelease.End, + if len(opts.SpotCheckJobSamples) == 0 && !opts.SampleRelease.Start.IsZero() && !opts.SampleRelease.End.IsZero() { + opts.SpotCheckJobSamples = []reqopts.SpotCheckJobSampleOpts{ + { + Name: "spotcheck-30d", + Release: reqopts.Release{ + Name: opts.SampleRelease.Name, + Start: opts.SampleRelease.Start, + End: opts.SampleRelease.End, + }, + IncludeVariants: map[string][]string{ + "JobTier": {"spotcheck-30d"}, + }, + }, } } diff --git a/pkg/apis/api/componentreport/crview/types.go b/pkg/apis/api/componentreport/crview/types.go index 4eb1bbe8e..a6b81e9e6 100644 --- a/pkg/apis/api/componentreport/crview/types.go +++ b/pkg/apis/api/componentreport/crview/types.go @@ -13,11 +13,10 @@ type View struct { VariantOptions reqopts.Variants `json:"variant_options" yaml:"variant_options"` AdvancedOptions reqopts.Advanced `json:"advanced_options" yaml:"advanced_options"` - // SpotCheckSample defines the sample window for spot-check job analysis. - // Spot-check jobs must pass at least once in this window (typically 30 days). - // If nil, spot-check analysis is disabled for this view. - // The release is inherited from SampleRelease; only the time window needs to be specified. - SpotCheckSample *SpotCheckWindow `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` + // SpotCheckJobSamples defines sample windows for spot-check job analysis, keyed by tier name + // (e.g. "spotcheck-30d"). Each entry specifies a time window and variant filters (ANDed) + // used to select which jobs to query. If empty, spot-check analysis is disabled for this view. + SpotCheckJobSamples map[string]SpotCheckJobSample `json:"spot_check_job_samples,omitempty" yaml:"spot_check_job_samples,omitempty"` Metrics Metrics `json:"metrics" yaml:"metrics"` RegressionTracking RegressionTracking `json:"regression_tracking" yaml:"regression_tracking"` @@ -40,9 +39,10 @@ type AutomateJira struct { Enabled bool `json:"enabled" yaml:"enabled"` } -// SpotCheckWindow defines just the time window for spot-check analysis. +// SpotCheckJobSample defines a spot-check sample window and variant filters for a tier. // The release is always inherited from SampleRelease. -type SpotCheckWindow struct { - RelativeStart string `json:"relative_start,omitempty" yaml:"relative_start,omitempty"` - RelativeEnd string `json:"relative_end,omitempty" yaml:"relative_end,omitempty"` +type SpotCheckJobSample struct { + RelativeStart string `json:"relative_start,omitempty" yaml:"relative_start,omitempty"` + RelativeEnd string `json:"relative_end,omitempty" yaml:"relative_end,omitempty"` + IncludeVariants map[string][]string `json:"include_variants,omitempty" yaml:"include_variants,omitempty"` } diff --git a/pkg/apis/api/componentreport/reqopts/types.go b/pkg/apis/api/componentreport/reqopts/types.go index 5a43297a5..9985044ad 100644 --- a/pkg/apis/api/componentreport/reqopts/types.go +++ b/pkg/apis/api/componentreport/reqopts/types.go @@ -19,10 +19,9 @@ type RequestOptions struct { CacheOption cache.RequestOptions TestFilters TestIDOptions []TestIdentification - // SpotCheckSample is the resolved sample window for spot-check job analysis. - // Only set when the view defines spot_check_sample. These jobs must pass at least - // once in this window (typically 30 days, longer than the normal 7-day test sample). - SpotCheckSample *Release `json:"spot_check_sample,omitempty" yaml:"spot_check_sample,omitempty"` + // SpotCheckJobSamples defines resolved sample windows for spot-check job analysis. + // Each entry specifies a tier name, time window, and variant filters to select jobs. + SpotCheckJobSamples []SpotCheckJobSampleOpts `json:"spot_check_job_samples,omitempty" yaml:"spot_check_job_samples,omitempty"` // ViewName is the name of the view used for this request, if any. // When generating test details URLs, if a view is present, we include just the view parameter // plus test-specific overrides, rather than expanding all view parameters into the URL. @@ -121,3 +120,10 @@ type Advanced struct { // When multiple key tests fail in the same job, only the highest priority (earliest in list) test is included. KeyTestNames []string `json:"key_test_names,omitempty" yaml:"key_test_names,omitempty"` } + +// SpotCheckJobSampleOpts is a resolved spot-check sample entry with absolute times. +type SpotCheckJobSampleOpts struct { + Name string `json:"name"` + Release `json:",inline" yaml:",inline"` //nolint:revive + IncludeVariants map[string][]string `json:"include_variants,omitempty" yaml:"include_variants,omitempty"` +} diff --git a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go index 3dd141470..c691a2700 100644 --- a/pkg/dataloader/regressioncacheloader/regressioncacheloader.go +++ b/pkg/dataloader/regressioncacheloader/regressioncacheloader.go @@ -505,18 +505,22 @@ func (l *RegressionCacheLoader) buildGenerator( TestFilters: view.TestFilters, } - if view.SpotCheckSample != nil { + for name, sample := range view.SpotCheckJobSamples { spotCheckRelative := reqopts.RelativeRelease{ Release: reqopts.Release{Name: view.SampleRelease.Name}, - RelativeStart: view.SpotCheckSample.RelativeStart, - RelativeEnd: view.SpotCheckSample.RelativeEnd, + RelativeStart: sample.RelativeStart, + RelativeEnd: sample.RelativeEnd, } resolved, err := utils.GetViewReleaseOptions( - l.releases, "spot_check", spotCheckRelative, cacheOpts.CRTimeRoundingFactor, cacheOpts.CRTimeRoundingOffset) + l.releases, "spot_check_"+name, spotCheckRelative, cacheOpts.CRTimeRoundingFactor, cacheOpts.CRTimeRoundingOffset) if err != nil { return nil, err } - reqOpts.SpotCheckSample = &resolved + reqOpts.SpotCheckJobSamples = append(reqOpts.SpotCheckJobSamples, reqopts.SpotCheckJobSampleOpts{ + Name: name, + Release: resolved, + IncludeVariants: sample.IncludeVariants, + }) } generator := componentreadiness.NewComponentReportGenerator( diff --git a/pkg/variantregistry/ocp.go b/pkg/variantregistry/ocp.go index 113677621..1e0fcf0c8 100644 --- a/pkg/variantregistry/ocp.go +++ b/pkg/variantregistry/ocp.go @@ -276,9 +276,9 @@ ORDER BY j.prowjob_job_name; // validateSpotCheckVariants returns an error if a job has JobTier=spotcheck without both // SpotCheckComponent and SpotCheckCapability defined. func validateSpotCheckVariants(jobName string, variants map[string]string) error { - if variants[VariantJobTier] == "spotcheck" { + if strings.HasPrefix(variants[VariantJobTier], "spotcheck-") { if variants[VariantSpotCheckComponent] == "" || variants[VariantSpotCheckCapability] == "" { - return fmt.Errorf("job %q has JobTier=spotcheck but is missing SpotCheckComponent or SpotCheckCapability", jobName) + return fmt.Errorf("job %q has JobTier=%s but is missing SpotCheckComponent or SpotCheckCapability", jobName, variants[VariantJobTier]) } } return nil @@ -798,9 +798,9 @@ func setSpotCheckClassification(_ logrus.FieldLogger, variants map[string]string // Note: blocking/informing/standard tiers may be downgraded to candidate by // adjustJobTierBasedOnView if the job's variants don't match the release-main view. func (v *OCPVariantLoader) setJobTier(_ logrus.FieldLogger, variants map[string]string, jobName string) { - // Jobs classified as spot-check get the spotcheck tier automatically. + // Jobs classified as spot-check get the spotcheck-30d tier automatically. if _, ok := variants[VariantSpotCheckComponent]; ok { - variants[VariantJobTier] = "spotcheck" + variants[VariantJobTier] = "spotcheck-30d" return } diff --git a/pkg/variantregistry/ocp_test.go b/pkg/variantregistry/ocp_test.go index 6859861a4..b685f8c16 100644 --- a/pkg/variantregistry/ocp_test.go +++ b/pkg/variantregistry/ocp_test.go @@ -2192,7 +2192,7 @@ func TestVariantSyncer(t *testing.T) { VariantSuite: "unknown", VariantUpgrade: VariantNoValue, VariantProcedure: "cpu-partitioning", - VariantJobTier: "spotcheck", + VariantJobTier: "spotcheck-30d", VariantAggregation: VariantNoValue, VariantSecurityMode: VariantDefaultValue, VariantFeatureSet: VariantDefaultValue, @@ -2222,7 +2222,7 @@ func TestVariantSyncer(t *testing.T) { VariantSuite: "etcd-scaling", VariantUpgrade: VariantNoValue, VariantProcedure: "etcd-scaling", - VariantJobTier: "spotcheck", + VariantJobTier: "spotcheck-30d", VariantAggregation: VariantNoValue, VariantSecurityMode: VariantDefaultValue, VariantFeatureSet: VariantDefaultValue, @@ -2608,13 +2608,13 @@ func TestAdjustJobTierBasedOnView(t *testing.T) { name: "spotcheck job is not adjusted even with non-matching variants", variants: map[string]string{ VariantRelease: "4.22", - VariantJobTier: "spotcheck", + VariantJobTier: "spotcheck-30d", VariantArch: "s390x", VariantPlatform: "rosa", VariantNetwork: "sdn", VariantOwner: "chaos", }, - expectedTier: "spotcheck", + expectedTier: "spotcheck-30d", }, { name: "job with no release is not adjusted", diff --git a/pkg/variantregistry/snapshot.yaml b/pkg/variantregistry/snapshot.yaml index 074d746d5..a0811ffa5 100644 --- a/pkg/variantregistry/snapshot.yaml +++ b/pkg/variantregistry/snapshot.yaml @@ -257033,7 +257033,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -257518,7 +257518,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -257739,7 +257739,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -258417,7 +258417,7 @@ periodic-ci-openshift-release-main-nightly-4.12-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -258912,7 +258912,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -258938,7 +258938,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -259399,7 +259399,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -259620,7 +259620,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -260400,7 +260400,7 @@ periodic-ci-openshift-release-main-nightly-4.13-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -261016,7 +261016,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -261042,7 +261042,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -261623,7 +261623,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -261868,7 +261868,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -262576,7 +262576,7 @@ periodic-ci-openshift-release-main-nightly-4.14-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -263294,7 +263294,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -263368,7 +263368,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -263925,7 +263925,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -264194,7 +264194,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -264902,7 +264902,7 @@ periodic-ci-openshift-release-main-nightly-4.15-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -265617,7 +265617,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -265691,7 +265691,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -266251,7 +266251,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -266541,7 +266541,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -267872,7 +267872,7 @@ periodic-ci-openshift-release-main-nightly-4.16-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -268729,7 +268729,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -268803,7 +268803,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-aws-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -269336,7 +269336,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-azure-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -269578,7 +269578,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -270934,7 +270934,7 @@ periodic-ci-openshift-release-main-nightly-4.17-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -271964,7 +271964,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -272038,7 +272038,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -272766,7 +272766,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -273152,7 +273152,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -275378,7 +275378,7 @@ periodic-ci-openshift-release-main-nightly-4.18-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -276487,7 +276487,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -276561,7 +276561,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -277217,7 +277217,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -277531,7 +277531,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -279849,7 +279849,7 @@ periodic-ci-openshift-release-main-nightly-4.19-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -281054,7 +281054,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -281128,7 +281128,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -281808,7 +281808,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -282146,7 +282146,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -284634,7 +284634,7 @@ periodic-ci-openshift-release-main-nightly-4.20-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -286007,7 +286007,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -286081,7 +286081,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -286935,7 +286935,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -287345,7 +287345,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -290103,7 +290103,7 @@ periodic-ci-openshift-release-main-nightly-4.21-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -291716,7 +291716,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -291838,7 +291838,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -292860,7 +292860,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -293294,7 +293294,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -296936,7 +296936,7 @@ periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -298669,7 +298669,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -298791,7 +298791,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -299765,7 +299765,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -300199,7 +300199,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -303745,7 +303745,7 @@ periodic-ci-openshift-release-main-nightly-4.23-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -305501,7 +305501,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-cpu-partitioning: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -305623,7 +305623,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -306795,7 +306795,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-azure-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -307229,7 +307229,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-gcp-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -310853,7 +310853,7 @@ periodic-ci-openshift-release-main-nightly-5.0-e2e-vsphere-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -342184,7 +342184,7 @@ periodic-ci-shiftstack-ci-release-4.14-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -342690,7 +342690,7 @@ periodic-ci-shiftstack-ci-release-4.16-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -343100,7 +343100,7 @@ periodic-ci-shiftstack-ci-release-4.17-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: runc FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -343585,7 +343585,7 @@ periodic-ci-shiftstack-ci-release-4.18-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -344094,7 +344094,7 @@ periodic-ci-shiftstack-ci-release-4.19-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -344603,7 +344603,7 @@ periodic-ci-shiftstack-ci-release-4.20-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -345112,7 +345112,7 @@ periodic-ci-shiftstack-ci-release-4.21-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default @@ -345621,7 +345621,7 @@ periodic-ci-shiftstack-ci-release-4.22-e2e-openstack-ovn-etcd-scaling: ContainerRuntime: crun FeatureSet: default Installer: ipi - JobTier: spotcheck + JobTier: spotcheck-30d LayeredProduct: none Network: ovn NetworkAccess: default From 6fa69630ffc040fcea940473fac01798b4439e55 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Wed, 17 Jun 2026 08:26:16 -0300 Subject: [PATCH 32/37] Defend in depth: double prevent sql injection via variant keys/values --- .../dataprovider/bigquery/provider.go | 12 ++++++------ .../dataprovider/bigquery/querygenerators.go | 3 ++- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 12c268fbe..55fa1324c 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -209,7 +209,7 @@ func (p *BigQueryProvider) QueryJobRuns(ctx context.Context, reqOptions reqopts. cleanV := param.Cleanse(v) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanV, cleanV, cleanV, v) + p.client.Dataset, cleanV, cleanV, cleanV, cleanV) } variantFilters := "" @@ -402,7 +402,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions cleanV := param.Cleanse(v) joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanV, cleanV, cleanV, v) + p.client.Dataset, cleanV, cleanV, cleanV, cleanV) selectVariants += fmt.Sprintf("jv_%s.variant_value AS variant_%s,\n", cleanV, cleanV) groupByParts = append(groupByParts, fmt.Sprintf("jv_%s.variant_value", cleanV)) } @@ -440,7 +440,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions if !joinedVariants.Has(group) { joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, cleanGroup) joinedVariants.Insert(group) } paramName := fmt.Sprintf("variantGroup_%s", cleanGroup) @@ -457,7 +457,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions if !joinedVariants.Has(group) { joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, cleanGroup) joinedVariants.Insert(group) } paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) @@ -582,7 +582,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO if !joinedVariants.Has(k) { joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanK, cleanK, cleanK, k) + p.client.Dataset, cleanK, cleanK, cleanK, cleanK) joinedVariants.Insert(k) } paramName := fmt.Sprintf("variant_%s", cleanK) @@ -599,7 +599,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO if !joinedVariants.Has(group) { joinVariants += fmt.Sprintf( "LEFT JOIN %s.job_variants jv_%s ON jobs.prowjob_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, group) + p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, cleanGroup) joinedVariants.Insert(group) } paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/querygenerators.go b/pkg/api/componentreadiness/dataprovider/bigquery/querygenerators.go index 5d2293827..62b1ef490 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/querygenerators.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/querygenerators.go @@ -396,8 +396,9 @@ func BuildComponentReportQuery( joinVariants := "" groupByVariants := "" for _, v := range sortedKeys(allJobVariants.Variants) { + cleanV := param.Cleanse(v) joinVariants += fmt.Sprintf("LEFT JOIN %s.job_variants jv_%s ON junit_data.variant_registry_job_name = jv_%s.job_name AND jv_%s.variant_name = '%s'\n", - client.Dataset, v, v, v, v) + client.Dataset, cleanV, cleanV, cleanV, cleanV) } for _, v := range reqOptions.VariantOption.DBGroupBy.List() { v = param.Cleanse(v) From be6bd82f9ee3b75013acb2aa7cb46d57129745d2 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Wed, 17 Jun 2026 08:34:41 -0300 Subject: [PATCH 33/37] defer spot-check sample resolution until after the sample release/date override --- .../utils/queryparamparser.go | 40 ++++++++++--------- 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/pkg/api/componentreadiness/utils/queryparamparser.go b/pkg/api/componentreadiness/utils/queryparamparser.go index 7882869b2..69c164424 100644 --- a/pkg/api/componentreadiness/utils/queryparamparser.go +++ b/pkg/api/componentreadiness/utils/queryparamparser.go @@ -55,24 +55,6 @@ func ParseComponentReportRequest( viewWarnings := api.ValidateVariants(allJobVariants, opts.VariantOption.IncludeVariants, " from view") warnings = append(warnings, viewWarnings...) } - - for name, sample := range view.SpotCheckJobSamples { - spotCheckRelative := reqopts.RelativeRelease{ - Release: reqopts.Release{Name: view.SampleRelease.Name}, - RelativeStart: sample.RelativeStart, - RelativeEnd: sample.RelativeEnd, - } - resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check_"+name, spotCheckRelative, crTimeRoundingFactor, crTimeRoundingOffset) - if resolveErr != nil { - err = resolveErr - return - } - opts.SpotCheckJobSamples = append(opts.SpotCheckJobSamples, reqopts.SpotCheckJobSampleOpts{ - Name: name, - Release: resolved, - IncludeVariants: sample.IncludeVariants, - }) - } } // Parse URL parameters - these override view defaults if view was provided @@ -190,6 +172,28 @@ func ParseComponentReportRequest( } } + // Resolve spot-check samples after all URL overrides (sampleRelease, date ranges) + // so they use the final release name and time context. + if view != nil { + for name, sample := range view.SpotCheckJobSamples { + spotCheckRelative := reqopts.RelativeRelease{ + Release: reqopts.Release{Name: opts.SampleRelease.Name}, + RelativeStart: sample.RelativeStart, + RelativeEnd: sample.RelativeEnd, + } + resolved, resolveErr := GetViewReleaseOptions(releases, "spot_check_"+name, spotCheckRelative, crTimeRoundingFactor, crTimeRoundingOffset) + if resolveErr != nil { + err = resolveErr + return + } + opts.SpotCheckJobSamples = append(opts.SpotCheckJobSamples, reqopts.SpotCheckJobSampleOpts{ + Name: name, + Release: resolved, + IncludeVariants: sample.IncludeVariants, + }) + } + } + // Params below this point can be used with and without views: // TODO: leave nil for safer cache keys if params not set, sync with metrics and primecache.go // TODO: unit test that metrics and primecache cache keys match a request object here From 5361a00b1449b4a068035ea3ed0a18a04e6c293b Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Wed, 17 Jun 2026 08:47:33 -0300 Subject: [PATCH 34/37] gocyclo fix --- .../dataprovider/bigquery/provider.go | 64 +++++++++++++++---- 1 file changed, 51 insertions(+), 13 deletions(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 55fa1324c..8489d302d 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -376,10 +376,22 @@ func (p *BigQueryProvider) LookupJobVariants(ctx context.Context, jobName string // --- SpotCheckQuerier --- -// spotCheckFallbackSQL provides a CASE expression for deriving component/capability +// spotCheckFallbackSQL provides COALESCE expressions for deriving component/capability // from job names when the SpotCheckComponent/SpotCheckCapability variants are not yet // populated in the job_variants table. This fallback can be removed once the variant // syncer has run and all spot-check jobs have the new variants. +const spotCheckComponentFallback = `COALESCE(jv_SpotCheckComponent.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'Node / Kubelet' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Etcd' + END)` + +const spotCheckCapabilityFallback = `COALESCE(jv_SpotCheckCapability.variant_value, + CASE + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%cpu-partitioning%%' THEN 'CPU Partitioning' + WHEN LOWER(jobs.prowjob_job_name) LIKE '%%etcd-scaling%%' THEN 'Scaling' + END)` + // QuerySpotCheckJobRuns queries the jobs table (not junit) for spot-check periodic/release // jobs, grouping by component, capability, and the DB group-by variants. // Returns aggregated pass/fail counts per group so the middleware can create @@ -451,7 +463,9 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions }) } - // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d) + // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d). + // During the transition period, also include 'rare' in the JobTier filter to match + // legacy jobs that haven't been reclassified yet. for _, group := range sortedKeys(spotCheckIncludeVariants) { cleanGroup := param.Cleanse(group) if !joinedVariants.Has(group) { @@ -460,19 +474,25 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, cleanGroup) joinedVariants.Insert(group) } + values := spotCheckIncludeVariants[group] + if group == "JobTier" { + values = append(values, "rare") + } paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) params = append(params, bigquery.QueryParameter{ Name: paramName, - Value: spotCheckIncludeVariants[group], + Value: values, }) } + // Use COALESCE fallback SQL to derive component/capability from job name substrings + // for legacy jobs that don't yet have the SpotCheckComponent/SpotCheckCapability variants. queryString := fmt.Sprintf(` SELECT %s - jv_SpotCheckComponent.variant_value AS spot_check_component, - jv_SpotCheckCapability.variant_value AS spot_check_capability, + %s AS spot_check_component, + %s AS spot_check_capability, COUNT(DISTINCT jobs.prowjob_build_id) AS total_runs, COUNT(DISTINCT IF(jobs.prowjob_state = 'success', jobs.prowjob_build_id, NULL)) AS successful_runs, ARRAY_AGG(DISTINCT jobs.prowjob_job_name) AS job_names, @@ -486,7 +506,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions %s GROUP BY %s, spot_check_component, spot_check_capability HAVING spot_check_component IS NOT NULL AND spot_check_capability IS NOT NULL - `, selectVariants, p.client.Dataset, joinVariants, variantFilters, + `, selectVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, + p.client.Dataset, joinVariants, variantFilters, groupByVariants) params = append(params, @@ -503,6 +524,18 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions return nil, fmt.Errorf("error executing spot check query: %w", err) } + results, err := parseSpotCheckRows(it, groupByVariantSet) + if err != nil { + return nil, err + } + + log.Infof("spot check query returned %d groups", len(results)) + return results, nil +} + +// parseSpotCheckRows reads spot-check aggregation results from a BigQuery iterator +// and returns them as SpotCheckGroup slices. +func parseSpotCheckRows(it *bigquery.RowIterator, groupByVariantSet sets.String) ([]dataprovider.SpotCheckGroup, error) { var results []dataprovider.SpotCheckGroup for { var rawRow map[string]bigquery.Value @@ -545,8 +578,6 @@ func (p *BigQueryProvider) QuerySpotCheckJobRuns(ctx context.Context, reqOptions } results = append(results, group) } - - log.Infof("spot check query returned %d groups", len(results)) return results, nil } @@ -593,7 +624,8 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO }) } - // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d) + // Apply spot-check sample-level include variant filters (e.g. JobTier: spotcheck-30d). + // During the transition period, also include 'rare' in the JobTier filter. for _, group := range sortedKeys(spotCheckIncludeVariants) { cleanGroup := param.Cleanse(group) if !joinedVariants.Has(group) { @@ -602,11 +634,15 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO p.client.Dataset, cleanGroup, cleanGroup, cleanGroup, cleanGroup) joinedVariants.Insert(group) } + values := spotCheckIncludeVariants[group] + if group == "JobTier" { + values = append(values, "rare") + } paramName := fmt.Sprintf("spotCheck_%s", cleanGroup) variantFilters += fmt.Sprintf(" AND (jv_%s.variant_value IN UNNEST(@%s))", cleanGroup, paramName) params = append(params, bigquery.QueryParameter{ Name: paramName, - Value: spotCheckIncludeVariants[group], + Value: values, }) } @@ -623,11 +659,13 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO AND jobs.prowjob_start < DATETIME(@To) AND jv_Release.variant_value = @Release AND (jobs.prowjob_job_name LIKE 'periodic-%%' OR jobs.prowjob_job_name LIKE 'release-%%') - AND LOWER(jv_SpotCheckComponent.variant_value) = LOWER(@SpotCheckComponent) - AND LOWER(jv_SpotCheckCapability.variant_value) = LOWER(@SpotCheckCapability) + AND LOWER(%s) = LOWER(@SpotCheckComponent) + AND LOWER(%s) = LOWER(@SpotCheckCapability) %s ORDER BY jobs.prowjob_start DESC - `, p.client.Dataset, joinVariants, variantFilters) + `, p.client.Dataset, joinVariants, + spotCheckComponentFallback, spotCheckCapabilityFallback, + variantFilters) params = append(params, bigquery.QueryParameter{Name: "From", Value: start}, From a912e88467d4fe8fe7bd3427812ef41024759f5c Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 19 Jun 2026 10:30:44 -0300 Subject: [PATCH 35/37] apm updatesn --- apm.lock.yaml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/apm.lock.yaml b/apm.lock.yaml index ae51ccba6..ac912064c 100644 --- a/apm.lock.yaml +++ b/apm.lock.yaml @@ -92,8 +92,8 @@ local_deployed_files: - .opencode/commands/sippy-update-ga-release-views.md - .opencode/commands/sippy-update-job-variant.md local_deployed_file_hashes: - .claude/commands/agentic-followup.md: sha256:3195764952aac75e75d85a861ec58d9f2cbb697735b9446eecbf50762241255d - .claude/commands/agentic-solve.md: sha256:cba8ca0bf1e1ee0d1175138e6648e6344d337615bf906909034b9a55b5531465 + .claude/commands/agentic-followup.md: sha256:d40313dcf0721911cae14a4af9774c021b0f16d24662a1b547524a5eef56ca86 + .claude/commands/agentic-solve.md: sha256:8df186e128d0ffac40f9494c8d46275ea12af43921981f05d52944ba072f40cd .claude/commands/sippy-dev-app.md: sha256:656276ed961940c137dde32ecdb0501427d4d811502a27125ba073adc770d266 .claude/commands/sippy-dev-frontend.md: sha256:42eae4b3bc610c9fcb43533a6fed229a6d1c409d279f3d6f93672986ede62e3a .claude/commands/sippy-dev-migrate.md: sha256:80160e88e0cc0fc09ab3dd9cc6fc496fe87dd8873800eb65d700868034d59da2 @@ -111,8 +111,8 @@ local_deployed_file_hashes: .claude/rules/general.md: sha256:997f68e86cb43485ec5f108be3417f9bbb43ae1faffd660d598f18260f5df3ce .claude/rules/mcp.md: sha256:ddfe965e7cf8cddbba1374c6ae582a20ac0af17c958bf10e1a4edff6ff2ad0b8 .claude/rules/testing.md: sha256:57834092f6732d17f8c1812d25290cfc1cfbbbeb6eae1561ba2240973c651866 - .cursor/commands/agentic-followup.md: sha256:3195764952aac75e75d85a861ec58d9f2cbb697735b9446eecbf50762241255d - .cursor/commands/agentic-solve.md: sha256:cba8ca0bf1e1ee0d1175138e6648e6344d337615bf906909034b9a55b5531465 + .cursor/commands/agentic-followup.md: sha256:d40313dcf0721911cae14a4af9774c021b0f16d24662a1b547524a5eef56ca86 + .cursor/commands/agentic-solve.md: sha256:8df186e128d0ffac40f9494c8d46275ea12af43921981f05d52944ba072f40cd .cursor/commands/sippy-dev-app.md: sha256:656276ed961940c137dde32ecdb0501427d4d811502a27125ba073adc770d266 .cursor/commands/sippy-dev-frontend.md: sha256:42eae4b3bc610c9fcb43533a6fed229a6d1c409d279f3d6f93672986ede62e3a .cursor/commands/sippy-dev-migrate.md: sha256:80160e88e0cc0fc09ab3dd9cc6fc496fe87dd8873800eb65d700868034d59da2 @@ -131,8 +131,8 @@ local_deployed_file_hashes: .cursor/rules/general.mdc: sha256:5bc6e1e12d53d85656248c9dc1239c74bcc0df29d5987f3b08e3d79e3df413b7 .cursor/rules/mcp.mdc: sha256:c02472afd46e4c89f71d4487dcd5da98b0c1bcbcf7f9cbc4d7ed4e7d3a206ec1 .cursor/rules/testing.mdc: sha256:e5ce80313a812750404d45355b462c2e3a6458f5bc20ad7996a5da1b169a4703 - .gemini/commands/agentic-followup.toml: sha256:cb017c493fb79dcfbc6eda8d2d058cd951a2aea700887e8754f0103858fd2089 - .gemini/commands/agentic-solve.toml: sha256:613044d7539a4550102760b434bad50fb8eaac76ba4a3721e7f6265dbbf2e8ed + .gemini/commands/agentic-followup.toml: sha256:81b2f8f0ddb3757d564bb4f35c4610d8cb466068a068296222b7fc01ef253977 + .gemini/commands/agentic-solve.toml: sha256:33a3850f284398fe4080c63e2501afbf2bd0fa2475fb922400111a8294e650df .gemini/commands/sippy-dev-app.toml: sha256:fc28174eeab4e440694a823bd838d429241997a018d8a13f32e0f67ca4d973c5 .gemini/commands/sippy-dev-frontend.toml: sha256:ec4ab5e1fb7581f09473e33b3ed4f53ce40509f23aac581e3b100ad1f59de5e5 .gemini/commands/sippy-dev-migrate.toml: sha256:25c98ba4bfdb95270dfcb4238ae688f9a66f0e645d8a4a6c5e03b1cf8db5cb7e @@ -143,8 +143,8 @@ local_deployed_file_hashes: .gemini/commands/sippy-generate-release-views.toml: sha256:9173d7507f469eda21a19f7e5ec47bce0c6ce87a73c1f5c10594a89417332927 .gemini/commands/sippy-update-ga-release-views.toml: sha256:662ac8d503d883cf6142c801b79e46616afb3d806e1d936e952d918b257ddf73 .gemini/commands/sippy-update-job-variant.toml: sha256:3b044dfaddeafa1a6d375918d5860785f8d315b593c355bbb7293a87c50361b8 - .opencode/commands/agentic-followup.md: sha256:3195764952aac75e75d85a861ec58d9f2cbb697735b9446eecbf50762241255d - .opencode/commands/agentic-solve.md: sha256:cba8ca0bf1e1ee0d1175138e6648e6344d337615bf906909034b9a55b5531465 + .opencode/commands/agentic-followup.md: sha256:d40313dcf0721911cae14a4af9774c021b0f16d24662a1b547524a5eef56ca86 + .opencode/commands/agentic-solve.md: sha256:8df186e128d0ffac40f9494c8d46275ea12af43921981f05d52944ba072f40cd .opencode/commands/sippy-dev-app.md: sha256:656276ed961940c137dde32ecdb0501427d4d811502a27125ba073adc770d266 .opencode/commands/sippy-dev-frontend.md: sha256:42eae4b3bc610c9fcb43533a6fed229a6d1c409d279f3d6f93672986ede62e3a .opencode/commands/sippy-dev-migrate.md: sha256:80160e88e0cc0fc09ab3dd9cc6fc496fe87dd8873800eb65d700868034d59da2 From e8a362e5454213d0785c3bb0b0ce9ff7813b0982 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 19 Jun 2026 11:01:09 -0300 Subject: [PATCH 36/37] Fix ordering of job runs on spot check regressions --- pkg/api/componentreadiness/dataprovider/bigquery/provider.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go index 8489d302d..15bdb1bb2 100644 --- a/pkg/api/componentreadiness/dataprovider/bigquery/provider.go +++ b/pkg/api/componentreadiness/dataprovider/bigquery/provider.go @@ -662,7 +662,7 @@ func (p *BigQueryProvider) QuerySpotCheckJobRunDetails(ctx context.Context, reqO AND LOWER(%s) = LOWER(@SpotCheckComponent) AND LOWER(%s) = LOWER(@SpotCheckCapability) %s - ORDER BY jobs.prowjob_start DESC + ORDER BY jobs.prowjob_start ASC `, p.client.Dataset, joinVariants, spotCheckComponentFallback, spotCheckCapabilityFallback, variantFilters) From 7ed7edcf39298021299112227d8be9cbcea832b1 Mon Sep 17 00:00:00 2001 From: Devan Goodwin Date: Fri, 19 Jun 2026 11:05:27 -0300 Subject: [PATCH 37/37] Omit spotcheck from normal component readiness queries --- config/views.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/config/views.yaml b/config/views.yaml index 87cd5b958..4e04a3253 100755 --- a/config/views.yaml +++ b/config/views.yaml @@ -47,7 +47,6 @@ component_readiness: - blocking - informing - standard - - spotcheck-30d LayeredProduct: - none - virt