Tenant metrics show only null values

[HaveFun83]

We currently evaluating the rustfs-operator for a minio replacement.

During our test deployment we found out that the rustfs-operator show only null values for the tenant metrics.

The polling itself looks healthy

example:

# HELP rustfs_operator_tenant_monitor_polls_total Total number of tenant storage monitor polls by result.
# TYPE rustfs_operator_tenant_monitor_polls_total counter
rustfs_operator_tenant_monitor_polls_total{result="error"} 1
rustfs_operator_tenant_monitor_polls_total{result="success"} 22
# HELP rustfs_operator_tenant_monitor_poll_duration_seconds Tenant storage monitor poll duration by result.
# TYPE rustfs_operator_tenant_monitor_poll_duration_seconds summary
rustfs_operator_tenant_monitor_poll_duration_seconds_count{result="error"} 1
rustfs_operator_tenant_monitor_poll_duration_seconds_sum{result="error"} 0.006699
rustfs_operator_tenant_monitor_poll_duration_seconds_count{result="success"} 22
rustfs_operator_tenant_monitor_poll_duration_seconds_sum{result="success"} 0.955364
# HELP rustfs_operator_tenant_storage_poll_success Whether the last tenant storage poll succeeded.
# TYPE rustfs_operator_tenant_storage_poll_success gauge
rustfs_operator_tenant_storage_poll_success{namespace="rustfs",tenant="rustfs"} 1.000000
# HELP rustfs_operator_tenant_storage_last_poll_timestamp_seconds Unix timestamp of the last tenant storage poll.
# TYPE rustfs_operator_tenant_storage_last_poll_timestamp_seconds gauge
rustfs_operator_tenant_storage_last_poll_timestamp_seconds{namespace="rustfs",tenant="rustfs"} 1785166926.000000
# HELP rustfs_operator_tenant_drives_online Number of RustFS drives reported online for a tenant.
# TYPE rustfs_operator_tenant_drives_online gauge
rustfs_operator_tenant_drives_online{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_drives_offline Number of RustFS drives reported offline for a tenant.
# TYPE rustfs_operator_tenant_drives_offline gauge
rustfs_operator_tenant_drives_offline{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_drives_healing Number of RustFS drives currently healing for a tenant.
# TYPE rustfs_operator_tenant_drives_healing gauge
rustfs_operator_tenant_drives_healing{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_raw_capacity_bytes Raw RustFS capacity in bytes for a tenant.
# TYPE rustfs_operator_tenant_raw_capacity_bytes gauge
rustfs_operator_tenant_raw_capacity_bytes{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_raw_used_bytes Raw RustFS used capacity in bytes for a tenant.
# TYPE rustfs_operator_tenant_raw_used_bytes gauge
rustfs_operator_tenant_raw_used_bytes{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_object_usage_bytes Object usage in bytes reported by RustFS for a tenant.
# TYPE rustfs_operator_tenant_object_usage_bytes gauge
rustfs_operator_tenant_object_usage_bytes{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_erasure_write_quorum_drives Estimated write quorum drive count implied by RustFS erasure parity.
# TYPE rustfs_operator_tenant_erasure_write_quorum_drives gauge
rustfs_operator_tenant_erasure_write_quorum_drives{namespace="rustfs",tenant="rustfs"} 0.000000
# HELP rustfs_operator_tenant_storage_healthy Whether tenant storage is online, not healing, and satisfies write quorum.
# TYPE rustfs_operator_tenant_storage_healthy gauge
rustfs_operator_tenant_storage_healthy{namespace="rustfs",tenant="rustfs"} 0.000000

rustfs-operator deploymentt via helm

console:
  enabled: false
  image:
    repository: <private-registry>/rustfs/operator
  podSecurityContext:
    fsGroup: null
  securityContext:
    runAsUser: null
operator:
  image:
    repository: <private-registry>/rustfs/operator
  leaderElect: null
  metrics:
    enabled: true
  podSecurityContext:
    fsGroup: null
  prometheusRule:
    enabled: true
  replicas: 1
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext:
    runAsUser: null
  serviceMonitor:
    enabled: true
  tenantMonitor:
    enabled: true
sts:
  enabled: false

tenant config

apiVersion: rustfs.com/v1alpha1
kind: Tenant
metadata:
  name: rustfs
  namespace: rustfs
spec:
  # Container image (optional - uses operator default if not specified)
  image: <private-registry>/rustfs/rustfs:1.0.0-beta.10

  pools:
    - name: dev-pool
      servers: 3
      persistence:
        volumesPerServer: 1 # Single PVC; RustFS uses no erasure parity

        # Optional: Reduce storage size for development
        volumeClaimTemplate:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi # Smaller for dev environments

  env:
    # Standard Rust logging (not RustFS-specific)
    - name: RUST_LOG
      value: "debug"
  credsSecret:
    name: rustfs
  securityContext:
    fsGroup: null
    runAsGroup: null
    runAsNonRoot: true
    runAsUser: null
  serviceAccountName: rustfs

  policies:
    - name: loki-bucket-policy
      document:
        configMapKeyRef:
          name: loki-bucket-policy
          key: policy.json

  users:
    - name: loki-user
      credsSecret:
        name: loki-user
      policies:
        - loki-bucket-policy

  buckets:
    - name: loki

The tenant itself look also healthy

Storage Information

Backend:        erasure
Disks:          3 (3 online, 0 offline)
Capacity:       1.55 GiB used / 11.15 GiB total

STATE      ENDPOINT                 PATH                 USED / TOTAL
online     http://rustfs-dev-pool-2.rustfs-hl.rustfs.svc.cluster.local:9000/data/rustfs0 /data/rustfs0        529.00 MiB / 3.72 GiB
online     http://rustfs-dev-pool-1.rustfs-hl.rustfs.svc.cluster.local:9000/data/rustfs0 /data/rustfs0        529.04 MiB / 3.72 GiB
online     http://rustfs-dev-pool-0.rustfs-hl.rustfs.svc.cluster.local:9000/data/rustfs0 /data/rustfs0        529.02 MiB / 3.72 GiB

Environment:
Openshift: 4.20.28

Maybe someone had an advice how we can troubleshoot this further?

[GatewayJ]

Thanks for the detailed report. We confirmed this is an Operator-side parsing bug and is unrelated to OpenShift or the Tenant storage health.

RustFS 1.0.0-beta.10 returns /rustfs/admin/v3/info using a response envelope:

{
  "info": {
    "usage": { ... },
    "backend": { ... },
    "pools": { ... }
  },
  "admin_discovery": { ... }
}

The Operator was deserializing the whole response directly as the inner info object. Because the relevant fields use Serde defaults, this shape mismatch did not produce an error: usage, backend, and pools silently became empty. The poll was then recorded as successful while all derived Tenant metrics were exported as zero. This explains the combination of tenant_storage_poll_success = 1 and zero drive/capacity values.

A fix has been prepared to deserialize the response envelope first and then use its required info field. The regression test now uses the real wrapped RustFS response shape. The change passes formatting, Clippy, the RustFS admin client test suite, and the Tenant storage metric test.

Until a release containing the fix is available, RustFS native monitoring can still be used for detailed data-plane metrics; the affected values are the Operator-derived Tenant summary metrics.

[HaveFun83]

Thanks a lot for the fix and detailed summary 🥇