Skip to content

refactor: deepen ETL infrastructure ownership#1463

Merged
zbnerd merged 45 commits into
developfrom
refactor/etl-infra-deepening
Jul 20, 2026
Merged

refactor: deepen ETL infrastructure ownership#1463
zbnerd merged 45 commits into
developfrom
refactor/etl-infra-deepening

Conversation

@zbnerd

@zbnerd zbnerd commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • Extract pipeline artifact identity, storage, and lifecycle ownership into module-pipeline-artifact.
  • Centralize Kafka delivery outcomes, retry/DLT handling, topology checks, and listener runtime in module-pipeline-messaging.
  • Move the valuation kernel into module-core and Nexon access into module-nexon-client.
  • Localize ETL executors, schedulers, subscriptions, and lifecycle wiring to external-api, calculator, synchronizer, and cleanup so their active runtime graphs no longer depend on module-infra.
  • Retain compatibility adapters and add source/dependency/runtime guards for the new boundaries.

Verification

  • Artifact identity guard, Kafka DLT topology tests, core dependency rules, and verifyActiveEtlInfraIsolation
  • External-api, calculator, and synchronizer subscription tests
  • Four worker Boot JAR builds
  • Four worker runtime smoke: actuator health UP, then graceful SIGTERM shutdown
  • Active worker source guards and git diff --check
  • Full repository test/root check (intentionally skipped to keep the refactor bounded)
  • Load/performance tests, Testcontainers suites, and live Nexon traffic (intentionally skipped for safety and runtime)

Review notes

  • The diff is large because the calculator-owned valuation CSV contains 413,802 rows and is relocated with the ownership change.
  • Compatibility adapters remain where downstream callers still need migration seams.
  • Runtime smoke used isolated Kafka consumer groups with external-api startup work disabled; no database reset or load test was performed.

zbnerd added 30 commits July 19, 2026 16:37
@zbnerd
zbnerd marked this pull request as ready for review July 20, 2026 22:56
@zbnerd
zbnerd merged commit cf85917 into develop Jul 20, 2026
4 of 5 checks passed
@zbnerd
zbnerd deleted the refactor/etl-infra-deepening branch July 22, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant