fix: track users reload separetely of server config#218
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses flakiness in config-reload behavior by ensuring the operator considers a replica “reloaded” only when both the server config and the users config have applied the same revision marker. It does this by adding a users-side reload marker (via a custom setting in an operator-owned settings profile) and by updating reconciliation logic to check both markers.
Changes:
- Add a custom users-profile setting (
custom_operator_reload_revision) that tracks the reload revision independently of the server config named collection revision. - Enable ClickHouse custom settings with
custom_settings_prefixes: custom_in the base server config template. - Update probe/reconcile logic to treat a replica as reloaded only when both the named-collection revision and the users-profile marker match the expected revision (plus update integration tests accordingly).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| internal/controller/clickhouse/templates/users.yaml.tmpl | Adds an operator settings profile with a custom setting to mark users reload revision; moves operator user onto that profile. |
| internal/controller/clickhouse/templates/base.yaml.tmpl | Enables custom_ custom settings so the users-profile marker can be applied/queried. |
| internal/controller/clickhouse/sync.go | Switches reload checks to require both server-config and users-config markers to match expected revision. |
| internal/controller/clickhouse/config.go | Adds constants and template params to wire the operator profile name + reload revision into generated users config. |
| internal/controller/clickhouse/commands.go | Extends replica probe to also read the users reload marker from system.settings_profile_elements and adds a Reloaded() helper. |
| internal/controller/clickhouse/commands_test.go | Updates integration test config and assertions to validate both reload markers are observed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Docs PR opened: ClickHouse/mintlify-docs-dev#144 Synced upstream ClickHouse Operator docs: added monitoring guide, External Secret section, AdditionalPort API type, and minor configuration and navigation updates. |
Why
ClickHouse updates users and server config in the background with the separate ConfigReloaders, so if server config is valid, but users are misconfigured, it will show a new revision in the named collection while failing on
SYSTEM RELOAD CONFIGThis makes the e2e test for config reload flaky
What
User custom config entry for the management user to track users' reload revision