Run the weekly comparator with deep error comparison in shadow (PR 7/8)#1532
Merged
Merged
Conversation
Turns deep error comparison on for the weekly CI run, in shadow mode: every Thrift-vs-SEA error divergence (class / SQLState / code / message / one-sided) is recorded in the report/CSV, but the run stays green. - run-comparator.sh and runJdbcComparator.yml default ERROR_COMPARISON_MODE to shadow (previously the deep comparison ran only when a caller set the flag). - ErrorPolicy is simplified to two modes, off and shadow. The authoritative (fail-the-build) mode is intentionally NOT included: enforcement over the DatabaseMetaData matrix can't be done cleanly at the diff-string level (CombinationExecutor collapses one-sided throws and infra failures into the same "Execution error: …" string), so a gate would either false-fail on infra flakes or silently miss real one-sided metadata divergences. Enforcement is better added later as its own change, driven by observed shadow-run data. - README documents the off/shadow modes and notes enforcement is deferred. Behavior: shadow produces the same report/CSV as before and never fails on a diff; nothing else changes. Exception-class comparison stays strict; message text is compared raw (no normalization). ErrorComparatorTest: 20/20. Isaac Review: 0 findings. NO_CHANGELOG=true (test-only comparator tooling). Co-authored-by: Isaac Signed-off-by: Sreekanth Vadigi <sreekanth.vadigi@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Turns deep error comparison on for the weekly CI run, in shadow mode: every Thrift-vs-SEA error divergence (class / SQLState / code / message / one-sided) is recorded in the report/CSV, but the run stays green. Final PR of the negative-cases series (builds on #1524–#1530).
run-comparator.shandrunJdbcComparator.ymldefaultERROR_COMPARISON_MODEtoshadow(previously the deep comparison only ran when a caller set the flag).ErrorPolicyis simplified to two modes:offandshadow.Why no
authoritative(fail-the-build) modeAn enforcement mode was prototyped and deliberately removed before this PR. Reason: enforcement over the
DatabaseMetaDatamatrix can't be done cleanly at the diff-string level —CombinationExecutorcollapses one-sided throws and infra failures into the same"Execution error: …"string, so a gate would either false-fail on an infra flake or silently miss a real one-sided metadata divergence. Rather than ship a gate with that known hole (dormant and unused, since we're not enforcing yet), enforcement is left as a clean future change, driven by the divergence data these shadow runs will accumulate.Behavior
Shadow produces the same report/CSV as before and never fails on a diff — nothing else changes. Exception-class comparison stays strict (exact match); message text is compared raw (no normalization).
Testing
ErrorComparatorTest— 20/20; compiles clean; dead-code check clean after removing the prototype gate; Isaac Review (/review --uncommitted) — 0 findings.NO_CHANGELOG=true (test-only comparator tooling, no driver behavior change).
This pull request and its description were written by Isaac.