docs(helm): worker-based instance health, with example update

docs/user/cluster-management/configuration.md

@@ -19,7 +19,7 @@ See below for descriptions of all available configuration options.
 | Priority Class                            | Set appropriate kubernetes priority class name for cluster agent and the operator pod. Additional details: [Priority Class](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)                                                                                                                                                                                                                                                                                                                                               |
 | Model Cache Volume Mount Options          | Configure the model cache volume mount options based on the CSI Driver capabilities on the cluster. Refer to the CSI Driver documentation. Defaults to `Enabled` and `ro,norecovery,nouuid` on an upgrade. Requires cluster reconfiguration after upgrade to prevent disruption.Additional details: [Mount options](https://man7.org/linux/man-pages/man8/mount.8.html)                                                                                                                                                                                                   |
 | Network CIDR Range                        | Quoted & comma separated list of CIDR range for outbound network access for the infrastructure components & workloads on the cluster.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
-| Worker Degradation Period                 | Stabilization time (in minutes) before cluster agent fails to consider a worker as healthy and initiates a purge.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| Worker Degradation Period                 | Stabilization time (in minutes) before cluster agent fails to consider a worker as healthy and initiates a purge. This also affects terminal worker failure timing for Helm functions that enable `StatusByWorkerReadiness`. See [Helm Functions](../helm-functions.md#use-worker-readiness-for-function-health).                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 
 ## Cluster Features
 

docs/user/helm-functions.md

@@ -39,7 +39,7 @@ All Pod specs in your helm chart will be updated with pull secrets at runtime, s
 
    - Important: The Helm chart name should not contain underscores or other special symbols, as that may cause issues during deployment.
 
-**Example Creation via API**
+### Example creation via API
 
 Please see our [sample helm chart used](https://github.com/NVIDIA/nvcf/tree/main/examples/function-samples/helmchart-samples/inference-test-sample) in this example for reference.
 
@@ -68,7 +68,8 @@ For gRPC-based functions, set `"inferenceURL" : "/gRPC"`. This signals to Cloud
 
 3. Proceed with function deployment and invocation normally.
 
-**Multi-node helm deployment**
+## Multi-node Helm deployment
+
 To create a multi-node helm deployment, you need to use the following format for the `instanceType`:
 `<CSP>.GPU.<GPU_NAME>_<number of gpus per node>x[.x<number of nodes>]`. For example, `DGXC.GPU.L40S_1x` is a single L40S instance while `ON-PREM.GPU.B200_8x.x2` is two full nodes of 8-way B200.
 
@@ -98,7 +99,7 @@ it will only do so if modification will not break your chart when it is installe
 Possible areas amenable to modification will be noted in the restrictions section below.
 Any standard that cannot be enforced by modification will result in error(s) during function creation.
 
-**Restrictions**
+#### Restrictions
 
 - Supported k8s artifacts under Helm Chart Namespace are listed below; others will be rejected:
   - ConfigMaps
@@ -167,3 +168,120 @@ curl -s -X POST "http://${GATEWAY_ADDR}/v2/nvcf/deployments/functions/${FUNCTION
      }]
  }'
 ```
+
+## Instance health
+
+Cloud Functions uses "holistic" Helm function health checking: on every reconcile,
+NVCA evaluates the health of every object applied from your Helm chart and their children
+(ex. a Deployment, its active ReplicaSet, and its replica Pods) plus infrastructure Pods.
+If any single object reaches a terminal bad state, the whole function instance is marked failed.
+NVCA then reports the failing object's status and events for debugging and re-creates the instance.
+
+An instance transitions through these phases:
+
+- Scheduling: one or more Pods have not yet been assigned to a node.
+- Starting: all Pods are scheduled and NVCA is waiting on readiness.
+- Running: all Helm chart objects and the worker Pod report healthy or ready.
+
+Because health depends on all chart objects, a Pod that is not the inference
+worker (for example a sidecar Deployment or an init Job) can fail the instance
+if it enters a terminal state. Use [`StatusByWorkerReadiness`](#use-worker-readiness-for-function-health)
+if only the inference worker's readiness should determine instance health.
+
+### Timeouts
+
+The following timeouts govern when a still-unhealthy object fails the instance.
+The clock for each timeout starts when the described condition is first
+observed, measured from the Pod launch time unless noted. Values are NVCA
+defaults. Only Worker Degradation Period is operator-configurable, through the
+`Worker Degradation Period` setting in [NVCA Configuration](./cluster-management/configuration.md).
+
+| Timeout | Default | Cause (when the clock starts) | Effect (when exceeded) |
+| --- | --- | --- | --- |
+| Pod scheduling | 10 minutes | A Pod cannot be scheduled onto a node, for example no node satisfies its resources or affinity. | The Pod is marked terminal (`PodStuckScheduling`) and the instance fails and is re-created. For tasks, the task's max queued duration is used instead of this default. |
+| Image pull error | 1 minute | A scheduled Pod reports `ErrImagePull` or `ImagePullBackOff` on any container or init container. | The Pod is marked terminal (`ImagePullIssues`) and the instance fails. Check registry credentials and image references. |
+| Container restart loop | 10 minutes | A container or init container restarts at least 3 times. | The Pod is marked terminal (stuck initializing, containers in restart loop) and the instance fails. |
+| Init container stuck | 2 hours | A Pod's init containers have not completed (the Pod is not yet `Initialized`). | The Pod is marked terminal (init container stuck) and the instance fails. |
+| Worker startup | 2 hours | A running Pod has not become ready during its initial startup, before it has ever reported ready. | The Pod is considered degraded and the instance fails. |
+| Worker Degradation Period | 30 minutes | A Pod that had been ready becomes not ready (containers report not ready) after being initialized. | The Pod is considered degraded, the instance is marked degraded, and NVCA kills and re-creates it. Operator-configurable. |
+| Pending timeout (max running) | 3 hours | Objects remain pending, that is not all objects reach ready, since the instance health condition first went unhealthy. | The instance fails with a pending timeout. |
+| Failing objects backoff | 90 seconds | An object reports a transient failure, for example `FailedMount` or `FailedAttachVolume`. | NVCA requeues and retries every 30 seconds for up to 90 seconds. If the object is still failing after 90 seconds, the instance fails. |
+
+Some conditions fail an instance immediately, without waiting for a timeout:
+
+- A Pod that enters the `Failed` phase, including admission rejection
+  (`UnexpectedAdmissionError`).
+- A Pod with restart policy `Never` whose non-restartable container terminates
+  with a non-zero exit code is treated as degraded right away, regardless of the
+  Worker Degradation Period.
+- A controller object (Deployment, ReplicaSet, StatefulSet, Job) that reports a
+  terminal condition such as `ProgressDeadlineExceeded`, a replica-failure
+  condition, or a failed Job.
+- A warning event that indicates a terminal error, such as `FailedCreate`,
+  `FailedUpdate`, `ReplicaSetCreateError`, or a `forbidden` error, on a tracked
+  object.
+
+### Use worker readiness for function health
+
+As described above, an unhealthy object, ex. a Pod, applied by the Helm chart can fail the function
+instance. This behavior may not be desirable if some objects in your chart are expected to fail or are not critical
+to serving inference. When your function is configured with the `StatusByWorkerReadiness` feature flag, the instance health check performed by NVCF becomes the sole determinant of instance health.
+The flag is read from the chart at install time to configure the function instance.
+
+Add `templates/nvcf-workload-config.yaml` to the function chart:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nvcf-workload-config
+data:
+  config.yaml: |
+    featureFlags:
+      StatusByWorkerReadiness: true
+```
+
+The ConfigMap **must** be named `nvcf-workload-config` and the `config.yaml` key **must** exist.
+Cloud Functions reads this configuration from the chart and does not create the ConfigMap
+in the instance namespace.
+
+Package and push the chart, then create and deploy the function:
+
+```bash
+helm package ./my-chart
+helm push ./my-chart-1.0.0.tgz \
+  oci://${REGISTRY}/${REPOSITORY}/charts
+
+./nvcf-cli function create \
+  --name my-helm-function \
+  --helm-chart \
+    oci://${REGISTRY}/${REPOSITORY}/charts/my-chart-1.0.0.tgz \
+  --helm-chart-service my-inference-service \
+  --inference-url /infer \
+  --inference-port 8000 \
+  --health-uri /health/ready \
+  --health-port 8000
+
+./nvcf-cli function deploy create \
+  --instance-type NCP.GPU.H100_1x \
+  --gpu H100 \
+  --min-instances 1 \
+  --max-instances 1
+```
+
+The function reaches `RUNNING` when the instance health endpoint `/health/ready` returns HTTP `200`,
+indicating readiness. With this flag enabled:
+
+- Other chart objects' statuses, including the inference Deployment and Service, are
+  informational only. Their statuses are sent in logs but do not gate `RUNNING`.
+- If a non-worker Pod goes down while the instance health endpoint reports ready, the
+  instance remains `RUNNING`. Cloud Functions reports the unhealthy object's status for
+  debugging, and Kubernetes can replace it.
+- If the health endpoint does not report ready, the instance is marked degraded
+  until it reports ready again or 30 minutes have passed (Worker Degradation Period, see [NVCA Configuration](./cluster-management/configuration.md)),
+  after which the instance is killed and re-created by NVCF.
+
+Without the ConfigMap, or with the flag set to `false`, Cloud Functions uses standard health behavior.
+
+See also the optional `statusByWorkerReadiness` value in
+[examples/function-samples/helmchart-samples/inference-test-sample](https://github.com/NVIDIA/nvcf/tree/main/examples/function-samples/helmchart-samples/inference-test-sample).

examples/function-samples/helmchart-samples/inference-test-sample/README.md

@@ -46,6 +46,11 @@ nvcf-cli registry add \
   --artifact-type CONTAINER
 ```
 
+To opt into worker-readiness-based instance health, set
+`statusByWorkerReadiness: true` in `inference-test/values.yaml`. This
+renders the optional `nvcf-workload-config` ConfigMap described in
+[Helm Functions](../../../../docs/user/helm-functions.md#use-worker-readiness-for-function-health).
+
 Create a function that references the chart, then deploy it:
 
 ```bash

examples/function-samples/helmchart-samples/inference-test-sample/inference-test/templates/nvcf-workload-config.yaml

@@ -0,0 +1,25 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if .Values.statusByWorkerReadiness }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nvcf-workload-config
+data:
+  config.yaml: |
+    featureFlags:
+      StatusByWorkerReadiness: true
+{{- end }}

examples/function-samples/helmchart-samples/inference-test-sample/inference-test/values.yaml

@@ -20,4 +20,8 @@ image:
   tag: "latest"
   pullPolicy: "IfNotPresent"
 
-inferencePort: 8000
+inferencePort: 8000
+
+# When true, render nvcf-workload-config so NVCA uses worker container readiness
+# for instance health instead of all chart objects. See docs/user/helm-functions.md.
+statusByWorkerReadiness: false

src/compute-plane-services/nvca/internal/miniservice/status.go

@@ -511,11 +511,6 @@ func (r *Reconciler) doStatusByWorkerReadiness(
 	}
 
 	// If the worker container is ready, then the miniservice is running.
-	/*
-		TODO:
-		* only use utils container readiness
-		* fail on utils container restart count > 3
-	*/
 	if workerContainerStatus.Ready {
 		ms.Status.Phase = v1alpha1.MiniServiceRunning
 		meta.SetStatusCondition(&ms.Status.Conditions, metav1.Condition{
@@ -526,7 +521,8 @@ func (r *Reconciler) doStatusByWorkerReadiness(
 		return reconcile.Result{}, nil
 	}
 
-	// The worker container is either progressing towards a ready state, or is in a terminal bad state.
+	// At this point the worker container is either progressing towards a ready state,
+	// or is in a terminal bad state.
 
 	var workerIssueReasons []string
 	if utilsStatus.AdditionalPodStatus != nil {