fix(dhcp): preserve strict readiness traffic gating for Kea pods#170
Open
dmathur0 wants to merge 1 commit into
Open
fix(dhcp): preserve strict readiness traffic gating for Kea pods#170dmathur0 wants to merge 1 commit into
dmathur0 wants to merge 1 commit into
Conversation
Separate DHCP probe responsibilities so an unconfigured pod can never become a Ready external DHCP target while still resolving the bootstrap dependency between Kea and the config-refresh process. - api: split liveness from readiness. Add /livez (Kea process/control channel alive only) for container liveness, and keep /healthcheck as a strict readiness gate (Kea online AND desired config applied). Remove the "no config in Redis -> OK" readiness exception so a bootstrap-only Kea is never Ready. Leave a clean seam/TODO where the Part 1 applied-hash-vs-expected-hash check plugs in. - kea: allow the endpoint to be overridden per-container via NV_CONFIG_MANAGER_KEA_SERVER / NV_CONFIG_MANAGER_KEA_PORT so the config-refresh deployment can reach Kea before pods are Ready without changing the shared INI. - auth: treat /livez as an unauthenticated probe path. - helm: point kea/healthcheck/api/config-sync liveness at /livez (a config mismatch no longer restarts a live Kea) while readiness stays on /healthcheck; add an internal-only *-kea-bootstrap Service exposing ONLY port 8000 with publishNotReadyAddresses: true; point config-refresh-v4 at it. The external DHCP LoadBalancer stays Ready-only and never exposes the bootstrap endpoint or the Kea control port. - tests: unit tests for strict readiness / livez semantics and the kea env override, plus helm rendering tests asserting the Service and probe invariants. Regenerate the DHCP OpenAPI spec for /livez. Signed-off-by: Davesh Mathur <daveshm@nvidia.com>
dmathur0
requested review from
jojung1,
polarweasel,
ryanheffernan,
spidercensus,
susanhooks and
zsblevins
as code owners
July 23, 2026 02:14
📝 WalkthroughWalkthroughThe DHCP API now separates Kea liveness from configuration readiness. Helm probes use ChangesDHCP probe behavior
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Kubelet
participant DHCPAPI
participant KeaClient
participant KeaControlChannel
Kubelet->>DHCPAPI: GET /livez
DHCPAPI->>KeaClient: check Kea status
KeaClient->>KeaControlChannel: status-get
KeaControlChannel-->>KeaClient: process status
KeaClient-->>DHCPAPI: liveness result
DHCPAPI-->>Kubelet: liveness response
Kubelet->>DHCPAPI: GET /healthcheck
DHCPAPI->>KeaClient: check status and applied config
KeaClient->>KeaControlChannel: status-get and config-get
KeaControlChannel-->>KeaClient: status and configuration
KeaClient-->>DHCPAPI: readiness result
DHCPAPI-->>Kubelet: readiness response
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8)deploy/helm/templates/dhcp.yamlTraceback (most recent call last): docs/api-specs/dhcp.openapi.jsonTraceback (most recent call last): Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/nv_config_manager/dhcp/api.py`:
- Around line 602-643: Update _assert_desired_config_applied so local/memfile
deployments no longer return immediately from the readiness check. Add an
equivalent validation that confirms the desired configuration has been applied
for those deployments, or explicitly document and track the intended exception
while preserving the strict unconfigured-pod behavior; keep the existing
PostgreSQL lease-database check for remote deployments.
In `@src/tests/helm/test_dhcp_services.py`:
- Around line 51-74: Update _render so only known environment-related Helm
failures are skipped; unexpected nonzero helm template results must fail the
test and include captured stderr in the failure. Preserve the existing skips for
missing Helm and unvendored dependencies, while removing the catch-all skip
behavior for template/rendering regressions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 50c963f1-960c-4705-bdef-5e81059d756b
📒 Files selected for processing (8)
deploy/helm/templates/dhcp.yamldocs/api-specs/dhcp.openapi.jsonsrc/nv_config_manager/common/auth.pysrc/nv_config_manager/dhcp/api.pysrc/nv_config_manager/dhcp/kea.pysrc/tests/dhcp/test_api.pysrc/tests/dhcp/test_kea.pysrc/tests/helm/test_dhcp_services.py
Comment on lines
+602
to
+643
| def _assert_desired_config_applied( | ||
| config: list[dict[str, Any]] | dict[str, Any], | ||
| app_config: ConfigParser, | ||
| ) -> None: | ||
| """Raise 500 unless Kea's running config reflects the desired configuration. | ||
|
|
||
| This is the *readiness* gate. It is intentionally strict: a freshly started | ||
| Kea running only the bootstrap config must NOT be reported ready, otherwise | ||
| an unconfigured pod could become an external DHCP target that silently drops | ||
| requests. There is deliberately no "empty Redis" escape hatch here -- the | ||
| bootstrap/validation path reaches Kea through the internal validation Service | ||
| (publishNotReadyAddresses), not by weakening readiness. | ||
|
|
||
| Today the applied-config signal is a proxy: on remote-lease-db deployments a | ||
| successful initial sync swaps Kea's ``lease-database`` from the bootstrap | ||
| ``memfile`` to ``postgresql``, so its presence means "the desired config was | ||
| applied". | ||
|
|
||
| TODO(part-1 hash verification): once ``KeaClient.get_config_hash`` / | ||
| ``expected_hash`` land, plug the applied-hash-vs-expected-hash comparison in | ||
| here (in addition to / in place of the lease-database heuristic) so readiness | ||
| reflects the exact desired configuration rather than a proxy signal. | ||
| """ | ||
| # KEA returns a list of responses, we want the first one | ||
| config_list = config if isinstance(config, list) else [config] | ||
|
|
||
| # Some error conditions don't return result or argument keys | ||
| if config_list[0].get("result", 0) != 0 or "arguments" not in config_list[0]: | ||
| error_msg = config_list[0].get("text", "No message provided") | ||
| raise HTTPException(status_code=500, detail=f"Failed to get KEA config: {error_msg}") | ||
|
|
||
| # Only remote (PostgreSQL) lease-db deployments have a config-applied proxy | ||
| # signal in the running Kea config. Local/memfile deployments have no such | ||
| # marker, so there is nothing further to assert here. | ||
| remote_lease_db = not app_config["dhcp.lease_db"].getboolean("local") | ||
| if not remote_lease_db: | ||
| return | ||
|
|
||
| dhcp4_config = config_list[0]["arguments"]["Dhcp4"] | ||
| lease_db_type = dhcp4_config.get("lease-database", {}).get("type", "memfile") | ||
| if lease_db_type != "postgresql": | ||
| raise HTTPException(status_code=500, detail="Lease database not present in Dhcp4 config") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Local/memfile deployments bypass the readiness gate
_assert_desired_config_applied returns immediately when remote_lease_db is false, so /healthcheck becomes ready for local/memfile deployments as soon as Kea is alive. If those deployments are meant to share the same “unconfigured pod stays unready” guarantee, this path needs an equivalent applied-config check or an explicit, tracked exception.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/nv_config_manager/dhcp/api.py` around lines 602 - 643, Update
_assert_desired_config_applied so local/memfile deployments no longer return
immediately from the readiness check. Add an equivalent validation that confirms
the desired configuration has been applied for those deployments, or explicitly
document and track the intended exception while preserving the strict
unconfigured-pod behavior; keep the existing PostgreSQL lease-database check for
remote deployments.
Comment on lines
+51
to
+74
| def _render(*set_args: str) -> str: | ||
| """Render the DHCP template with values-ci.yaml, or skip if unrenderable.""" | ||
| if shutil.which("helm") is None: | ||
| pytest.skip("helm binary not available") | ||
| if not (_CHART_DIR / "charts").is_dir(): | ||
| pytest.skip("helm chart dependencies not vendored (run `helm dependency build`)") | ||
|
|
||
| cmd = [ | ||
| "helm", | ||
| "template", | ||
| _RELEASE, | ||
| str(_CHART_DIR), | ||
| "--values", | ||
| str(_CHART_DIR / "values-ci.yaml"), | ||
| "--show-only", | ||
| _TEMPLATE, | ||
| ] | ||
| for pair in set_args: | ||
| cmd += ["--set", pair] | ||
|
|
||
| result = subprocess.run(cmd, capture_output=True, text=True, check=False) | ||
| if result.returncode != 0: | ||
| pytest.skip(f"helm template failed (environment issue): {result.stderr.strip()}") | ||
| return result.stdout |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Catch-all skip on nonzero helm template exit code can hide real template regressions.
Lines 72-73 skip the test for any nonzero exit code, lumping genuine template syntax/rendering errors in with the two already-explicit environment checks (missing helm binary, unvendored chart deps). Since this file exists specifically to guard the readiness-gating invariants added by this PR, a broken template (e.g. Helm control-flow, bad value reference) would silently skip instead of failing CI — defeating the purpose of the test suite.
Narrow the catch-all so unexpected failures surface as test failures (with the captured stderr) rather than skips.
♻️ Suggested fix
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
- if result.returncode != 0:
- pytest.skip(f"helm template failed (environment issue): {result.stderr.strip()}")
+ if result.returncode != 0:
+ raise AssertionError(f"helm template failed unexpectedly:\n{result.stderr.strip()}")
return result.stdout📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _render(*set_args: str) -> str: | |
| """Render the DHCP template with values-ci.yaml, or skip if unrenderable.""" | |
| if shutil.which("helm") is None: | |
| pytest.skip("helm binary not available") | |
| if not (_CHART_DIR / "charts").is_dir(): | |
| pytest.skip("helm chart dependencies not vendored (run `helm dependency build`)") | |
| cmd = [ | |
| "helm", | |
| "template", | |
| _RELEASE, | |
| str(_CHART_DIR), | |
| "--values", | |
| str(_CHART_DIR / "values-ci.yaml"), | |
| "--show-only", | |
| _TEMPLATE, | |
| ] | |
| for pair in set_args: | |
| cmd += ["--set", pair] | |
| result = subprocess.run(cmd, capture_output=True, text=True, check=False) | |
| if result.returncode != 0: | |
| pytest.skip(f"helm template failed (environment issue): {result.stderr.strip()}") | |
| return result.stdout | |
| def _render(*set_args: str) -> str: | |
| """Render the DHCP template with values-ci.yaml, or skip if unrenderable.""" | |
| if shutil.which("helm") is None: | |
| pytest.skip("helm binary not available") | |
| if not (_CHART_DIR / "charts").is_dir(): | |
| pytest.skip("helm chart dependencies not vendored (run `helm dependency build`)") | |
| cmd = [ | |
| "helm", | |
| "template", | |
| _RELEASE, | |
| str(_CHART_DIR), | |
| "--values", | |
| str(_CHART_DIR / "values-ci.yaml"), | |
| "--show-only", | |
| _TEMPLATE, | |
| ] | |
| for pair in set_args: | |
| cmd += ["--set", pair] | |
| result = subprocess.run(cmd, capture_output=True, text=True, check=False) | |
| if result.returncode != 0: | |
| raise AssertionError(f"helm template failed unexpectedly:\n{result.stderr.strip()}") | |
| return result.stdout |
🧰 Tools
🪛 ast-grep (0.44.1)
[error] 70-70: Use of unsanitized data to create processes
Context: subprocess.run(cmd, capture_output=True, text=True, check=False)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(os-system-unsanitized-data)
[error] 70-70: Command coming from incoming request
Context: subprocess.run(cmd, capture_output=True, text=True, check=False)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/tests/helm/test_dhcp_services.py` around lines 51 - 74, Update _render so
only known environment-related Helm failures are skipped; unexpected nonzero
helm template results must fail the test and include captured stderr in the
failure. Preserve the existing skips for missing Helm and unvendored
dependencies, while removing the catch-all skip behavior for template/rendering
regressions.
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ensures an unconfigured DHCP pod never becomes a Ready external DHCP target. A freshly started Kea on the bootstrap config can't offer a lease but can still receive and drop a request that should have gone to a configured replica. Previously
/healthcheckreturnedOKwhen Redis had no desired config (so the refresh process could reach Kea forconfig-test), which let a bootstrap-only pod become Ready — conflicting with strict traffic gating.This splits probe responsibilities and resolves the bootstrap dependency explicitly.
Changes
/livez(liveness) asserts only that Kea is online viastatus-get. A config mismatch can no longer restart a live Kea; a dead Kea → 500 → kubelet recycles the container. Added to unauthenticated probe paths incommon/auth.py./healthcheck(readiness) is now strict: Kea online AND desired config applied. Removed the "empty Redis → OK" escape hatch, so a bootstrap-only pod is never Ready. Applied-config proxy islease-database.type == postgresql(bootstrap usesmemfile), with aTODO(part-1 hash verification)seam in_assert_desired_config_applied(...)where Part 1'sget_config_hash/expected_hashcheck plugs in.<dhcp>-kea-bootstrapClusterIP exposing only port 8000 withpublishNotReadyAddresses: true.config-refresh-v4points at it viaNV_CONFIG_MANAGER_KEA_SERVER/PORT(honored byKeaClient.from_config). The external DHCP LoadBalancer stays Ready-only and never exposes 8000.kea/healthcheck/api/config-sync-v4liveness →/livez;healthcheck/apireadiness stay/healthcheck.docs/api-specs/dhcp.openapi.jsonfor/livez.Test plan
/livezready / offline / timeoutpublishNotReadyAddresses& exposes only 8000; internal Ready-only; refresh env → bootstrap; liveness/livezvs readiness/healthcheck--checkup to datePart 3 of 5 from the DHCP config-sync recovery proposal, opened independently against
main.Merge note
Touches
deploy/helm/templates/dhcp.yaml(also touched by Parts 2 & 5) and leaves aTODO(part-1 hash verification)seam for #168 (Part 1). Whichever merges later will need a small rebase on thedhcp.yamlprobes and the applied-config check.Summary by CodeRabbit
New Features
/livezendpoint for process liveness checks, separate from configuration readiness.Improvements