Design Spec: vast-cluster-health
Parent: #123
Target: rw-cli-codecollection
Spec
codebundle_name: "vast-cluster-health"
target_collection: "rw-cli-codecollection"
display_name: "VAST Data Cluster Health"
author: "rw-codebundle-agent"
purpose: |
Monitor VAST Data cluster-wide health via the VMS REST API and Prometheus
exporter endpoints. Detects degraded cluster state, capacity exhaustion,
hardware failures on CNodes/DNodes, and cluster-level performance bottlenecks
that affect all tenants and clients (Kubernetes, NFS, block, S3).
tasks:
- name: "Check VMS Cluster Health Status for Cluster `${VAST_CLUSTER_NAME}`"
description: "Queries /api/prometheusmetrics/vms_state and VMS cluster status to detect DEGRADED (0) vs CLUSTERED (1) state and any active cluster alerts"
script_name: "check-vms-cluster-health.sh"
expected_issue_severity: [1, 2]
access_level: "read-only"
data_type: "metrics"
- name: "Check Cluster Capacity Utilization for Cluster `${VAST_CLUSTER_NAME}`"
description: "Evaluates physical and logical capacity utilization from /api/prometheusmetrics/all and capacity endpoints; raises issues when usage exceeds CAPACITY_THRESHOLD percent"
script_name: "check-cluster-capacity.sh"
expected_issue_severity: [2, 3]
access_level: "read-only"
data_type: "metrics"
- name: "Check CNode and DNode Hardware Health for Cluster `${VAST_CLUSTER_NAME}`"
description: "Inspects CNode/DNode state, SSD/SCM health, and hardware fault indicators from Prometheus exporter metrics"
script_name: "check-node-hardware-health.sh"
expected_issue_severity: [2, 3]
access_level: "read-only"
data_type: "metrics"
- name: "Check Cluster Degraded Components and Active Alerts for Cluster `${VAST_CLUSTER_NAME}`"
description: "Lists degraded boxes, failed drives, offline nodes, and active VMS alerts that indicate partial cluster failure"
script_name: "check-degraded-components.sh"
expected_issue_severity: [1, 2]
access_level: "read-only"
data_type: "logs-config"
- name: "Analyze Cluster Protocol Performance for Cluster `${VAST_CLUSTER_NAME}`"
description: "Reviews cluster-wide IOPS, bandwidth, and latency by storage protocol (NFS, block, S3) to detect IO stalls or abnormal drops in data flow"
script_name: "analyze-cluster-performance.sh"
expected_issue_severity: [3, 4]
access_level: "read-only"
data_type: "metrics"
- name: "Check Replication and Protection Group Status for Cluster `${VAST_CLUSTER_NAME}`"
description: "Verifies replication links, protection groups, and snapshot policies are healthy and not blocking writes or causing capacity pressure"
script_name: "check-replication-status.sh"
expected_issue_severity: [2, 3]
access_level: "read-only"
data_type: "config"
scope:
level: "Resource"
qualifiers:
- VAST_CLUSTER_NAME
- VAST_VMS_ENDPOINT
iteration_pattern: |
One SLX per VAST cluster registered in the workspace. User provides
VAST_VMS_ENDPOINT (VMS API URL) and VAST_CLUSTER_NAME. When RESOURCES
is "All", iterate over all clusters discovered via the VMS /clusters/
endpoint; otherwise iterate over the comma-separated cluster list.
resource_types:
- "vast_data_cluster"
generation_strategy: |
One SLX per VAST cluster. Qualifier: cluster name. Resource match:
vast_data_cluster. Discovery script queries VMS /clusters/ REST endpoint
using VAST_VMS_ENDPOINT credentials. SLX baseName: vast-cluster-health.
env_vars:
- name: VAST_VMS_ENDPOINT
description: "VMS REST API base URL (e.g. https://vms.example.com)"
required: true
- name: VAST_CLUSTER_NAME
description: "VAST cluster display name for scoping and issue titles"
required: true
- name: RESOURCES
description: "Cluster name(s) or 'All' for auto-discovery via VMS API"
required: false
default: "All"
- name: CAPACITY_THRESHOLD
description: "Physical/logical capacity utilization percent that triggers an issue"
required: false
default: "85"
- name: CRITICAL_CAPACITY_THRESHOLD
description: "Critical capacity threshold percent"
required: false
default: "95"
secrets:
- name: vast_vms_credentials
description: "VMS API authentication credentials"
format: |
JSON object with: USERNAME, PASSWORD
Alternative: API_TOKEN (Bearer token if supported by VMS version)
platform:
name: "vast_data"
cli_tools:
- "curl"
- "jq"
- "python3"
auth_methods:
- "VMS username/password (vast_vms_credentials)"
- "VMS API token (vast_vms_credentials.API_TOKEN)"
api_docs: "https://kb.vastdata.com/documentation/docs/exporting-metrics-to-prometheus"
related_bundles:
- name: "k8s-pvc-healthcheck"
relationship: "complements"
notes: "k8s-pvc-healthcheck covers Kubernetes PVC binding and mount utilization; this bundle covers the VAST backend cluster health"
- name: "azure-storage-health"
relationship: "complements"
notes: "Azure storage health pattern for capacity and misconfiguration checks; different platform"
- name: "vast-tenant-storage-health"
relationship: "complements"
notes: "Tenant-level capacity, QoS, and view health; this bundle covers cluster-wide infrastructure"
- name: "vast-k8s-csi-health"
relationship: "complements"
notes: "Kubernetes CSI driver and PVC-to-VAST tracing; this bundle covers native VMS cluster metrics"
test_scenarios:
- name: "healthy_cluster"
description: "Cluster in CLUSTERED state with capacity below threshold and all nodes healthy"
expected_issues: 0
- name: "degraded_cluster"
description: "VMS reports DEGRADED state with offline DNode"
expected_issues: 2
expected_severities: [1, 2]
- name: "capacity_pressure"
description: "Logical capacity above CAPACITY_THRESHOLD with no hardware faults"
expected_issues: 1
expected_severities: [3]
notes: |
VAST exposes Prometheus-format metrics via VMS REST API at paths like
/api/prometheusmetrics/vms_state, /api/prometheusmetrics/all,
/api/prometheusmetrics/cnodes, /api/prometheusmetrics/dnodes. Scripts
should parse plain-text Prometheus exposition format. VMS REST API docs
are available at {VAST_VMS_ENDPOINT}/docs. Prefer REST + curl over
requiring a local Prometheus instance. Gracefully degrade individual
tasks when optional metrics endpoints are unavailable on older VAST
versions. Do not scope to individual CNode/DNode instances — cluster
level only per RunWhen scoping rules.
Design Spec: vast-cluster-health
Parent: #123
Target:
rw-cli-codecollectionSpec