Skip to content

spark and commons upgrade #91

Merged
MatloaItumeleng merged 5 commits intomasterfrom
feature/74-spark3-upgrade
Apr 20, 2026
Merged

spark and commons upgrade #91
MatloaItumeleng merged 5 commits intomasterfrom
feature/74-spark3-upgrade

Conversation

@MatloaItumeleng
Copy link
Copy Markdown
Contributor

@MatloaItumeleng MatloaItumeleng commented Apr 8, 2026

Release notes:

  • Upgrades Spark 2.4 with Spark 3.

    • Scala crossbuild focus Scala 2.12 and Scala 2.13.
    • target Java 1.11

  • Upgrades spark-commons to compatible version.

closes #74

ABLL526
ABLL526 reviewed Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ABLL526 ABLL526 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but the checks are failing. I will approve once that is fixed.

@MatloaItumeleng MatloaItumeleng marked this pull request as draft April 9, 2026 08:45
@MatloaItumeleng
Copy link
Copy Markdown
Contributor Author

MatloaItumeleng commented Apr 9, 2026

Few things picked up addressing them seems Spark’s generated expression's has changed on some tests e.g for representation of an empty array literal

  • expected: ... ELSE [[]] ...
  • actual: ... ELSE [ARRAY()] ...

@MatloaItumeleng
handling spark expressions error for empty arrary, changed null case …
9fb2231 
…& Java 11 normalised ClassCastException message format
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 10, 2026

JaCoCo code coverage report - scala 2.12.12

Metric (instruction) Coverage Threshold Status
Overall 81.57% 80.0% ✅
Changed Files 0.0% 80.0% ✅
Report Coverage (O/Ch) Threshold (O/Ch) Status (O/Ch)
spark-data-standardization Jacoco Report - scala:2.12.20 81.57% / 0.0% 80.0% / 80.0% ✅/✅
File Path Coverage Threshold Status

No changed file in reports.

@MatloaItumeleng MatloaItumeleng marked this pull request as ready for review April 10, 2026 11:30
@ABLL526
Copy link
Copy Markdown
Contributor

ABLL526 commented Apr 10, 2026 •
edited
Loading

@MatloaItumeleng Looks all good. Nice work finding those errors and fixing them. It still seems there is one test for Spark 2.11.12 this might need to be removed as well. I will approve once completed.

dk1844
dk1844 previously approved these changes Apr 13, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@dk1844 dk1844 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment for the cast message normalization.
Also is there a project (even a sample one) that uses this library that would be able to test-use the updated spark version to check the correctness? (just a thought)

Comment thread src/test/scala/za/co/absa/standardization/validation/field/ScalarFieldValidatorSuite.scala Outdated
ABLL526
ABLL526 previously approved these changes Apr 13, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ABLL526 ABLL526 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good

@MatloaItumeleng MatloaItumeleng dismissed stale reviews from ABLL526 and dk1844 via d823048 April 14, 2026 09:17
dk1844
dk1844 reviewed Apr 15, 2026
View reviewed changes
Comment thread src/test/scala/za/co/absa/standardization/validation/field/ScalarFieldValidatorSuite.scala Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 17, 2026

JaCoCo code coverage report - scala 2.12.20

Metric (instruction) Coverage Threshold Status
Overall 81.57% 80.0% ✅
Changed Files 0.0% 80.0% ✅
Report Coverage (O/Ch) Threshold (O/Ch) Status (O/Ch)
spark-data-standardization Jacoco Report - scala:2.12.20 81.57% / 0.0% 80.0% / 80.0% ✅/✅
File Path Coverage Threshold Status

No changed file in reports.

dk1844
dk1844 approved these changes Apr 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@dk1844 dk1844 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ABLL526
ABLL526 approved these changes Apr 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@ABLL526 ABLL526 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MatloaItumeleng MatloaItumeleng merged commit bbc354e into master Apr 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dk1844 dk1844 dk1844 approved these changes

@ABLL526 ABLL526 ABLL526 approved these changes

@Zejnilovic Zejnilovic Awaiting requested review from Zejnilovic Zejnilovic is a code owner

@benedeki benedeki Awaiting requested review from benedeki benedeki is a code owner

@lsulak lsulak Awaiting requested review from lsulak lsulak is a code owner

@yruslan yruslan Awaiting requested review from yruslan yruslan is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Replace Spark 2.4 with Spark 3 support

3 participants

@MatloaItumeleng @ABLL526 @dk1844