Conversation
…project structure
…it repo syncs required two reconciles to reflect the latest commit
… a warning when using insecureSkipVerify with repositories, updated examples
…toreKey to reflect new namespaced convention
* feat: add pre-upgrade database migration job support
Introduces an opt-in Kubernetes Job that runs Liquibase schema migrations
before Gateway pods roll out during an upgrade. Gateway pods are blocked
until the job succeeds, preventing pods from starting against an unmigrated
schema.
Key behaviors:
- Works with Gateway 11.2.2 and later which provides four schema update modes in entrypoint.sh: default, skip,
liquibase-only, liquibase-only-with-unlock
- Migration job enabled via spec.app.management.database.migrationJob.enabled
- clearLocks field releases stale Liquibase locks before migrating
- Gateway pods automatically use skip mode when migration job is enabled
- Job is auto-replaced when spec changes (image, jdbcUrl, clearLocks)
- Failed jobs block the deployment and require manual deletion to retry
- jdbcUrl override only applies in diskless mode; node.properties wins
in non-diskless mode, consistent with Helm behavior
* feat: add pre-upgrade database migration job support
Introduces an opt-in Kubernetes Job that runs Liquibase schema migrations
before Gateway pods roll out during an upgrade. Gateway pods are blocked
until the job succeeds, preventing pods from starting against an unmigrated
schema.
Key behaviors:
- Works with Gateway 11.2.2 and later which provides four schema update modes in entrypoint.sh: default, skip,
liquibase-only, liquibase-only-with-unlock
- Migration job enabled via spec.app.management.database.migrationJob.enabled
- clearLocks field releases stale Liquibase locks before migrating
- Gateway pods automatically use skip mode when migration job is enabled
- Job is auto-replaced when spec changes (image, jdbcUrl, clearLocks)
- Failed jobs block the deployment and require manual deletion to retry
- jdbcUrl override only applies in diskless mode; node.properties wins
in non-diskless mode, consistent with Helm behavior
* fix bugs. Improve error handling.
* fix to modify / characters in branch names to be replaced with hyphen.
* Replace job-existence-based migration tracking with persistent status in
Gateway.Status.MigrationStatus. This addresses issue with previous
implementation would re-run migrations on every 12-hour reconcile if the
completed Job had been deleted.
Changes:
- Add MigrationStatus struct (SpecHash, Complete) to GatewayStatus in
gateway_types.go. Once Complete is true, GatewayMigrationJob returns
immediately on all future reconciles regardless of Job existence.
- Add DeepCopyInto/DeepCopy for MigrationStatus in zz_generated.deepcopy.go.
- Update CRD schema (security.brcmlabs.com_gateways.yaml) to include the
new migrationStatus status fields.
- Rewrite GatewayMigrationJob in reconcile/migration_job.go:
- Compute a 16-char spec hash from image, effective jdbcUrl, clearLocks,
and activeDeadlineSeconds. A hash change (e.g. image upgrade) resets
status and triggers a fresh migration automatically.
- On first enable: write hash to status, create Job, wait.
- On Job success: write Complete=true to status, unblock Deployment.
- On Job failure (both pod attempts exhausted): log error with exact
kubectl delete command, block Deployment. User deletes Job to retry.
- On disabled: clean up any orphaned Job.
- Remove migrationJobSpecChanged — spec change detection is now fully
handled by the hash comparison.
Recovery after failure: fix the root cause (run kubectl patch with clearLocks:true or
restore DB to pre-upgrade state if partially migrated) , then:
kubectl delete job <name>-db-migration -n <namespace>
The operator automatically creates a new Job. No kubectl apply needed.
* improve re-use of the MigrationJobName function.
* Gate on migration status instead of the GatewayMigrationJob
* Remove extra comments.
* Fix bug.
* Added control-plane related End2End tests for the database migration.
* Migration job: run-to-completion reconcile, event-driven waits, pod spec parity
- Reconcile loop no longer short-circuits while the migration job is pending — it runs every op to completion each pass and only stops on a genuine error. The Deployment step alone gates on Gateway.Status.MigrationStatus.
- GatewayMigrationJob returns nil for all "not done yet" states instead of a synthetic ErrMigrationPending; progress is driven by the Job's own watch events (create/update/delete) rather than a fixed poll interval.
- Fixed an edge case where a spec change (e.g. image upgrade) with no existing Job to delete could stall migration Job recreation for up to 12 hours — it now creates the replacement Job in the same reconcile pass when there's nothing to wait on.
- Failure log message no longer embeds a kubectl command.
- The migration Job's pod/container spec now inherits the same settings as the main Gateway Deployment (security contexts, resources, node selector, affinity, tolerations, topology spread constraints, pod annotations/labels) instead of a bare-minimum hand-built spec, and uses the same app.kubernetes.io/* labeling convention as other operator-managed resources.
- Added a regression test for the "spec changed, no existing Job" case; existing migration job tests updated for the new nil-return contract.
* Fix to accept forward slash in the branch name.
* Updated gateway.md with Gateway.spec.app.management.database.migrationJob
* Fix jenkins to allow branch name to have forward slash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gateway Configuration
General Updates
Resolved issues
Known issues