-
Notifications
You must be signed in to change notification settings - Fork 38
feat(self-managed): wire autoscaler observability #384
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
c062011
d2c52ed
8a6db33
6fb7f83
f05d143
44af5ef
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| #!/bin/sh | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # https://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| set -eu | ||
|
|
||
| chart_dir=${CHART_DIR:-.} | ||
| release_name=${RELEASE_NAME:-function-autoscaler} | ||
| namespace=${NAMESPACE:-nvcf} | ||
| tmpdir=$(mktemp -d) | ||
|
|
||
| cleanup() { | ||
| rm -rf "$tmpdir" | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| export HELM_CONFIG_HOME=${HELM_CONFIG_HOME:-"$tmpdir/helm-config"} | ||
| export HELM_CACHE_HOME=${HELM_CACHE_HOME:-"$tmpdir/helm-cache"} | ||
| export HELM_DATA_HOME=${HELM_DATA_HOME:-"$tmpdir/helm-data"} | ||
|
|
||
| render() { | ||
| helm template "$release_name" "$chart_dir" \ | ||
| --namespace "$namespace" \ | ||
| --values "$chart_dir/values.yaml" \ | ||
| --set functionautoscaler.image.repository=rs-autoscaler \ | ||
| "$@" | ||
| } | ||
|
|
||
| assert_contains() { | ||
| file=$1 | ||
| pattern=$2 | ||
|
|
||
| if ! grep -Fq -- "$pattern" "$file"; then | ||
| echo "expected rendered output to contain: $pattern" >&2 | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| assert_not_contains() { | ||
| file=$1 | ||
| pattern=$2 | ||
|
|
||
| if grep -Fq -- "$pattern" "$file"; then | ||
| echo "expected rendered output not to contain: $pattern" >&2 | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| default_render="$tmpdir/default.yaml" | ||
| render > "$default_render" | ||
| assert_contains "$default_render" "secrets.json.tmpl: |-" | ||
| assert_not_contains "$default_render" "nvcfApiToken" | ||
| assert_not_contains "$default_render" "NVCF_API_TOKEN_BLOCK" | ||
| assert_not_contains "$default_render" "services/nvcf-api/jwt/sign/nvcf-autoscaler-service" | ||
|
|
||
| static_token_render="$tmpdir/static-token.yaml" | ||
| render --set functionautoscaler.vault.staticNvcfApiToken.enabled=true > "$static_token_render" | ||
| assert_contains "$static_token_render" "services/nvcf-api/jwt/sign/nvcf-autoscaler-service" | ||
| assert_contains "$static_token_render" '"nvcfApiToken": "{{ .Data.token }}",' | ||
| assert_not_contains "$static_token_render" "NVCF_API_TOKEN_BLOCK" | ||
|
|
||
| custom_path_render="$tmpdir/custom-path.yaml" | ||
| render \ | ||
| --set functionautoscaler.vault.staticNvcfApiToken.enabled=true \ | ||
| --set functionautoscaler.vault.staticNvcfApiToken.path=services/custom/jwt/sign/autoscaler \ | ||
| > "$custom_path_render" | ||
| assert_contains "$custom_path_render" "services/custom/jwt/sign/autoscaler" | ||
| assert_not_contains "$custom_path_render" "services/nvcf-api/jwt/sign/nvcf-autoscaler-service" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,26 +45,35 @@ controlPlane: | |
| selector: | ||
| app.kubernetes.io/instance: state-metrics | ||
| app.kubernetes.io/name: helm-nvcf-state-metrics | ||
| - name: function-autoscaler | ||
| enabled: true | ||
| selector: | ||
| app.kubernetes.io/instance: function-autoscaler | ||
| app.kubernetes.io/name: helm-nvcf-function-autoscaler | ||
| - name: grpc-proxy | ||
| enabled: true | ||
| selector: | ||
| app.kubernetes.io/instance: grpc-proxy | ||
| - name: api | ||
| enabled: true | ||
| path: /actuator/prometheus | ||
| port: management-port | ||
| selector: | ||
| app.kubernetes.io/name: helm-nvcf-api | ||
| - name: sis | ||
| enabled: true | ||
| namespaces: | ||
| - sis | ||
| path: /actuator/prometheus | ||
| port: management-port | ||
| selector: | ||
| app.kubernetes.io/name: helm-nvcf-sis | ||
| - name: llm-api-gateway | ||
| enabled: true | ||
| selector: | ||
| app.kubernetes.io/instance: llm-api-gateway | ||
| # Off until the Invocation Service exporter is enabled. | ||
| - name: invocation-service | ||
| enabled: false | ||
| enabled: true | ||
| selector: | ||
| app.kubernetes.io/instance: invocation-service | ||
| app.kubernetes.io/name: helm-nvcf-invocation-service | ||
|
Comment on lines
75
to
+79
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C3 'invocation-service|name:\s*metrics|targetPort:\s*metrics|/metrics' deploy/helm deploy/stacks/self-managedRepository: NVIDIA/nvcf Length of output: 43866 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect the invocation-service chart for its metrics service/endpoint contract
printf '\n== values ==\n'
sed -n '130,165p' deploy/helm/http-invocation/values.yaml
printf '\n== deployment ==\n'
sed -n '1,140p' deploy/helm/http-invocation/templates/deployment.yaml
printf '\n== service ==\n'
sed -n '1,120p' deploy/helm/http-invocation/templates/service.yaml
printf '\n== default monitor values ==\n'
sed -n '60,95p' deploy/stacks/observability/charts/nvcf-default-monitors/values.yamlRepository: NVIDIA/nvcf Length of output: 7163 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '\n== default monitor templates ==\n'
fd -a . deploy/stacks/observability/charts/nvcf-default-monitors/templates
printf '\n== monitor template references ==\n'
rg -n -C3 'path:|port:|ServiceMonitor|PodMonitor|invocation-service' deploy/stacks/observability/charts/nvcf-default-monitors/templates deploy/stacks/observability/charts/nvcf-default-monitors/values.yamlRepository: NVIDIA/nvcf Length of output: 9066 Add a metrics Service port before enabling the invocation-service monitor.
🤖 Prompt for AI Agents |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 4962
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 16110
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 266
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 257
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 50368
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 15848
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 11698
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 42627
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 19568
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 7266
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 5941
Retarget
apito an existing Service port. TheapiService only exposeshttp, soport: management-portwon’t match unless that Service adds a namedmanagement-portport.🤖 Prompt for AI Agents