diff --git a/README.md b/README.md index 9d0d3e94..6e300202 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ NVIDIA Fleet Intelligence Agent - Host agent for GPU telemetry collection and attestation. -Built on top of [leptonai/gpud](https://github.com/leptonai/gpud) +For complete documentation on NVIDIA Fleet Intelligence, including the service dashboard, enrollment, and user guide, see the [NVIDIA Fleet Intelligence Documentation](https://docs.nvidia.com/fleet-intelligence/latest/index.html). ## Overview @@ -33,15 +33,13 @@ Built on top of [leptonai/gpud](https://github.com/leptonai/gpud) ## Documentation -**Important:** Documentation links are relative to the branch or tag you are viewing. The default GitHub view uses `main`, which may describe unreleased changes. When installing or upgrading a specific agent version, switch to that version's release tag first. - -- [Helm Installation](docs/install-helm.md) - Kubernetes (Helm) installation and troubleshooting -- [DEB Installation](docs/install-deb.md) - Ubuntu package install, update, and uninstall -- [RPM Installation](docs/install-rpm.md) - RHEL/Rocky/Alma/Amazon package install, update, and uninstall -- [Architecture](docs/architecture.md) - Bare metal and Kubernetes architecture, dependencies, and runtime flow -- [Usage](docs/usage.md) - Commands, HTTP API, integration, and troubleshooting -- [Configuration](docs/configuration.md) - Environment variables and service configuration -- [Development](docs/development.md) - Building from source and contributing +- [Helm Installation](https://docs.nvidia.com/fleet-intelligence/latest/agent/install-helm.html) - Kubernetes (Helm) installation and troubleshooting +- [DEB Installation](https://docs.nvidia.com/fleet-intelligence/latest/agent/install-deb.html) - Ubuntu package install, update, and uninstall +- [RPM Installation](https://docs.nvidia.com/fleet-intelligence/latest/agent/install-rpm.html) - RHEL/Rocky/Alma/Amazon package install, update, and uninstall +- [Architecture](https://docs.nvidia.com/fleet-intelligence/latest/agent/architecture.html) - Bare metal and Kubernetes architecture, dependencies, and runtime flow +- [Usage](https://docs.nvidia.com/fleet-intelligence/latest/agent/usage.html) - Commands, HTTP API, integration, and troubleshooting +- [Configuration](https://docs.nvidia.com/fleet-intelligence/latest/agent/configuration.html) - Environment variables and service configuration +- [Development](https://docs.nvidia.com/fleet-intelligence/latest/agent/development.html) - Building from source and contributing ## Contributing diff --git a/docs/architecture.md b/docs/architecture.md deleted file mode 100644 index 02874dcb..00000000 --- a/docs/architecture.md +++ /dev/null @@ -1,93 +0,0 @@ -# Architecture - -This document explains how the Fleet Intelligence Agent works at runtime on bare metal and Kubernetes. - -## Agent runtime model - -At runtime, the agent follows a consistent pipeline: - -1. Discover available components (GPU/DCGM and system-level checks). -2. Collect telemetry and health signals. -3. Evaluate component health and status. -4. Publish results through: - - HTTP API - - Prometheus metrics endpoint (`/metrics`) - - Optional remote export - - Optional local file export - -## Bare metal runtime architecture - -On bare metal, the agent runs directly on the host and talks to local NVIDIA stack dependencies. - -### Dependency graph - -```mermaid -flowchart LR - GPU["NVIDIA GPUs"] --> DRV["NVIDIA Driver"] --> NVML["NVML"] - NVML --> DCGM["DCGM HostEngine"] - DCGM --> AGENT["fleetint agent process"] - - AGENT --> COMP["Component checks + health evaluation"] - COMP --> API["HTTP API + /metrics"] - COMP --> EXPORT["Export pipeline"] -``` - -### Runtime behavior - -- GPU/DCGM components read health and telemetry from DCGM/NVML-backed data paths. -- System components read host state (CPU, memory, disk, network, OS/kernel/library/PCI views). -- The agent evaluates component health continuously and serves current state via API/metrics. -- Exporters serialize telemetry/events/health data for downstream ingestion. - -## Kubernetes runtime architecture - -In Kubernetes, the same agent runtime executes inside a pod (typically one pod per GPU node). - -### Dependency graph - -```mermaid -flowchart LR - subgraph CL["Kubernetes cluster"] - subgraph NODE["GPU node"] - GPU["NVIDIA GPUs"] --> DRV["NVIDIA Driver"] --> NVML["NVML"] - - subgraph GOP["GPU platform services"] - DCGM["DCGM HostEngine"] - SVC["DCGM Service endpoint"] - end - - subgraph POD["fleet-intelligence-agent pod"] - AGENT["fleetint runtime"] - COMP["Component checks + health evaluation"] - API["HTTP API + /metrics"] - end - end - end - - DCGM --> SVC --> AGENT - AGENT --> COMP --> API -``` - -### Runtime behavior - -- Agent component logic is the same as bare metal. -- DCGM-backed components consume DCGM over the cluster endpoint (`DCGM_URL`). -- The pod reads required host-level views through mounted paths and runtime-provided device access. -- Health evaluation and export behavior remain identical to host mode. - -## Data flow inside the agent - -```mermaid -flowchart TD - INPUT["GPU + DCGM + System signals"] --> COLLECT["Collectors"] - COLLECT --> STATE["Component state model"] - STATE --> HEALTH["Health evaluation"] - HEALTH --> API["HTTP/metrics output"] - HEALTH --> EXPORT["Exporters (remote/file)"] -``` - -## Dependency and failure behavior - -- If DCGM is unavailable, DCGM-backed GPU components degrade while non-DCGM components can continue. -- If GPU runtime/device access is unavailable in Kubernetes, GPU-facing checks may fail or report degraded state. -- If upstream signals recover, component health transitions back based on normal collection/evaluation cycles. diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index 5078fe20..00000000 --- a/docs/configuration.md +++ /dev/null @@ -1,207 +0,0 @@ -# Configuration - -This page documents the runtime configuration for `fleetint run`: -- configurable environment variables -- configurable `fleetint run` flags -- how to set them on bare metal and in Kubernetes - -## Where to configure - -### Bare Metal - -Package installs run `fleetint` through the `fleetintd` systemd service: - -```ini -ExecStart=/usr/bin/fleetint run $FLEETINT_FLAGS -``` - -Configure runtime settings in `/etc/default/fleetint`, then restart the service: - -```bash -sudo systemctl restart fleetintd -``` - -- Set environment variables directly in `/etc/default/fleetint` -- Set `fleetint run` flags through `FLEETINT_FLAGS="..."` - -### Kubernetes - -The Helm chart configures the container entrypoint as `fleetint run` and exposes: - -- environment variables under `env.*` -- common `run` flags through dedicated chart values such as `logLevel`, `listenAddress`, and `components` - -Apply changes by updating `values.yaml` or using `helm upgrade --set ...`. - -## Configurable Environment Variables - -These environment variables are read by `fleetint run` at startup. - -| Environment variable | Description | Default | Bare metal | Kubernetes | -| --- | --- | --- | --- | --- | -| `DCGM_URL` | DCGM HostEngine address used by the agent for DCGM-backed components. | bare metal: `localhost`, Helm chart: `nvidia-dcgm.gpu-operator.svc:5555` | `/etc/default/fleetint` | `env.DCGM_URL` | -| `DCGM_URL_IS_UNIX_SOCKET` | Treat `DCGM_URL` as a Unix socket path instead of a network address. | `false` | `/etc/default/fleetint` | `env.DCGM_URL_IS_UNIX_SOCKET` | -| `MALLOC_ARENA_MAX` | glibc arena cap to constrain RSS growth for DCGM/cgo-heavy workloads. | `4` | `/etc/default/fleetint` | `env.MALLOC_ARENA_MAX` | -| `FLEETINT_COLLECT_INTERVAL` | Export interval for health data. Valid range: `1s` to `24h`. | `1m` | `/etc/default/fleetint` | `env.FLEETINT_COLLECT_INTERVAL` | -| `FLEETINT_INCLUDE_METRICS` | Include metrics data in export payloads. | `true` | `/etc/default/fleetint` | `env.FLEETINT_INCLUDE_METRICS` | -| `FLEETINT_INCLUDE_EVENTS` | Include event data in export payloads. | `true` | `/etc/default/fleetint` | `env.FLEETINT_INCLUDE_EVENTS` | -| `FLEETINT_INCLUDE_MACHINEINFO` | Include machine information in export payloads. | `true` | `/etc/default/fleetint` | `env.FLEETINT_INCLUDE_MACHINEINFO` | -| `FLEETINT_INCLUDE_HEALTHCHECKS` | Include component data and health details in export payloads. | `true` | `/etc/default/fleetint` | `env.FLEETINT_INCLUDE_HEALTHCHECKS` | -| `FLEETINT_METRICS_LOOKBACK` | Lookback window for metrics included in each export. | `1m` | `/etc/default/fleetint` | `env.FLEETINT_METRICS_LOOKBACK` | -| `FLEETINT_EVENTS_LOOKBACK` | Lookback window for events included in each export. | `1m` | `/etc/default/fleetint` | `env.FLEETINT_EVENTS_LOOKBACK` | -| `FLEETINT_CHECK_INTERVAL` | Health check interval for monitored components. Valid range: `1s` to `24h`. | `1m` | `/etc/default/fleetint` | `env.FLEETINT_CHECK_INTERVAL` | -| `FLEETINT_RETRY_MAX_ATTEMPTS` | Maximum retry attempts for failed exports. Minimum: `0`. | `3` | `/etc/default/fleetint` | `env.FLEETINT_RETRY_MAX_ATTEMPTS` | -| `FLEETINT_INVENTORY_ENABLED` | Enable or disable the inventory loop. | `true` | `/etc/default/fleetint` | `env.FLEETINT_INVENTORY_ENABLED` | -| `FLEETINT_INVENTORY_INTERVAL` | Inventory loop interval override. Minimum: `5m`. | `1h` | `/etc/default/fleetint` | `env.FLEETINT_INVENTORY_INTERVAL` | -| `FLEETINT_ATTESTATION_ENABLED` | Enable or disable the attestation loop. | `true` | `/etc/default/fleetint` | `env.FLEETINT_ATTESTATION_ENABLED` | -| `FLEETINT_ATTESTATION_INTERVAL` | Attestation interval override. Minimum: `5m`. | `24h` | `/etc/default/fleetint` | `env.FLEETINT_ATTESTATION_INTERVAL` | -| `HTTP_PROXY` | Proxy URL for outbound HTTP requests. | empty | `/etc/default/fleetint` | `env.HTTP_PROXY` | -| `HTTPS_PROXY` | Proxy URL for outbound HTTPS requests. | empty | `/etc/default/fleetint` | `env.HTTPS_PROXY` | - -Notes: - -- Duration-valued environment variables use Go duration syntax such as `30s`, `1m`, `10m`, or `24h`. -- These environment variables modify the telemetry exporter configuration and runtime loop intervals used by `fleetint run`. -- `DCGM_URL` and `DCGM_URL_IS_UNIX_SOCKET` configure connectivity to DCGM HostEngine for DCGM-backed components. - -### Bare Metal Example - -```bash -sudoedit /etc/default/fleetint -``` - -```bash -FLEETINT_FLAGS="--log-level=info --components=all,-accelerator-nvidia-dcgm-prof" -DCGM_URL="localhost" -DCGM_URL_IS_UNIX_SOCKET="false" -MALLOC_ARENA_MAX="4" -FLEETINT_COLLECT_INTERVAL="2m" -FLEETINT_INCLUDE_EVENTS="false" -FLEETINT_CHECK_INTERVAL="30s" -HTTPS_PROXY="http://proxy.example.com:3128" -``` - -```bash -sudo systemctl restart fleetintd -``` - -### Kubernetes Example - -```yaml -logLevel: info -listenAddress: 0.0.0.0:15133 -retentionPeriod: 24h -components: all,-accelerator-nvidia-dcgm-prof - -env: - DCGM_URL: "nvidia-dcgm.gpu-operator.svc:5555" - DCGM_URL_IS_UNIX_SOCKET: "false" - MALLOC_ARENA_MAX: "4" - FLEETINT_COLLECT_INTERVAL: "2m" - FLEETINT_INCLUDE_EVENTS: "false" - FLEETINT_CHECK_INTERVAL: "30s" - FLEETINT_INVENTORY_INTERVAL: "5m" - HTTPS_PROXY: "http://proxy.example.com:3128" -``` - -Apply with: - -```bash -helm upgrade --install fleet-intelligence-agent \ - oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - -n \ - -f values.yaml -``` - -## Configurable `fleetint run` Flags - -These are the `fleetint run` flags supported by the CLI. - -| Flag | Description | Default | Bare metal | Kubernetes | -| --- | --- | --- | --- | --- | -| `--log-level` | Log level: `debug`, `info`, `warn`, `error`. | unset by CLI; packaged bare-metal default is `warn` via `FLEETINT_FLAGS` | `FLEETINT_FLAGS="--log-level=..."` | `logLevel` | -| `--log-file` | Log file path. Leave empty to log to stdout/stderr. | empty | `FLEETINT_FLAGS="--log-file=..."` | not exposed by chart by default | -| `--listen-address` | Listen address for the agent API server. An absolute path creates a Unix socket; a `host:port` value opens a TCP listener. | `/run/fleetint/fleetint.sock` | `FLEETINT_FLAGS="--listen-address=..."` | `listenAddress` | -| `--retention-period` | Retention period for stored metrics and events. Minimum `1m`. | `24h` | `FLEETINT_FLAGS="--retention-period=..."` | `retentionPeriod` | -| `--components` | Comma-separated component selection. Use `all`, `*`, explicit names, and `-name` exclusions. | empty flag value, which means enable all components by default | `FLEETINT_FLAGS="--components=..."` | `components` | -| `--offline-mode` | Disable the HTTP API server and write telemetry to files instead. | `false` | `FLEETINT_FLAGS="--offline-mode ..."` | not exposed by chart by default | -| `--path` | Absolute path to the output directory for offline mode. Must not point inside restricted system directories. Required with `--offline-mode`. | empty | `FLEETINT_FLAGS="--path=/path ..."` | not exposed by chart by default | -| `--duration` | Offline-mode collection duration in `HH:MM:SS` format. Required with `--offline-mode`. | empty | `FLEETINT_FLAGS="--duration=00:05:00 ..."` | not exposed by chart by default | -| `--format` | Offline-mode output format: `json` or `csv`. | `json` | `FLEETINT_FLAGS="--format=csv ..."` | not exposed by chart by default | -| `--enable-fault-injection` | Enable the local fault-injection endpoint for testing. | `false` | `FLEETINT_FLAGS="--enable-fault-injection"` | not exposed by chart by default | - -## Component Selection - -Use `--components` to control which monitoring components are enabled. - -Examples: - -```bash -# Enable all components -fleetint run --components=all - -# Enable only specific components -fleetint run --components=accelerator-nvidia-dcgm-thermal,accelerator-nvidia-dcgm-utilization,cpu,memory - -# Start from the default set and disable one component -fleetint run --components=all,-accelerator-nvidia-dcgm-prof -``` - -Rules: - -- `all`, `*`, or an empty component list enables all components. -- `all,-` starts with all components, then disables specific ones. -- An explicit comma-separated list enables only the named components. -- A non-matching explicit value effectively disables all components. - -Available component names: - -**NVIDIA GPU components** - -- `accelerator-nvidia-infiniband` -- `accelerator-nvidia-nccl` -- `accelerator-nvidia-peermem` -- `accelerator-nvidia-persistence-mode` -- `accelerator-nvidia-error-sxid` -- `accelerator-nvidia-error-xid` - -**NVIDIA GPU DCGM components** - -- `accelerator-nvidia-dcgm-clock` -- `accelerator-nvidia-dcgm-cpu` -- `accelerator-nvidia-dcgm-inforom` -- `accelerator-nvidia-dcgm-mem` -- `accelerator-nvidia-dcgm-nvlink` -- `accelerator-nvidia-dcgm-nvswitch` -- `accelerator-nvidia-dcgm-pcie` -- `accelerator-nvidia-dcgm-power` -- `accelerator-nvidia-dcgm-prof` -- `accelerator-nvidia-dcgm-thermal` -- `accelerator-nvidia-dcgm-utilization` - -**System components** - -- `cpu` -- `disk` -- `memory` -- `network-ethernet` -- `os` -- `library` - -## Verify Effective Configuration - -### Bare Metal - -```bash -sudo cat /etc/default/fleetint -sudo systemctl status fleetintd -sudo journalctl -u fleetintd -f -``` - -### Kubernetes - -```bash -helm get values fleet-intelligence-agent -n -kubectl get daemonset fleet-intelligence-agent -n -o yaml -kubectl logs -n -l app.kubernetes.io/name=fleet-intelligence-agent --tail=100 -``` diff --git a/docs/development.md b/docs/development.md deleted file mode 100644 index 27ffc528..00000000 --- a/docs/development.md +++ /dev/null @@ -1,194 +0,0 @@ -# Development - -## Setup - -### Clone Repository - -```bash -git clone https://github.com/NVIDIA/fleet-intelligence-agent.git -cd fleet-intelligence-agent -``` - -### Prerequisites - -- **Go**: 1.26.4 -- **GoReleaser**: For building packages (install from [goreleaser.com](https://goreleaser.com/install/)) -- **For ARM64 cross-compilation** (optional): `gcc-aarch64-linux-gnu` and `g++-aarch64-linux-gnu` - -### Build - -```bash -# Build the binary -make fleetint - -# Build all targets -make all - -# Run tests -make test -``` - -## Project Structure - -``` -fleetint/ -├── cmd/ -│ └── fleetint/ # Main CLI application -├── internal/ -│ ├── cmdutil/ # Command utilities -│ ├── config/ # Configuration management -│ ├── exporter/ # Data export functionality -│ │ ├── collector/ # Data collection -│ │ ├── converter/ # Format conversion (CSV, OTLP) -│ │ └── writer/ # Output writers (file, HTTP) -│ ├── machineinfo/ # Machine information gathering -│ ├── server/ # HTTP API server -│ └── version/ # Version information -├── deployments/ -│ └── packages/ # Package build configurations -│ ├── scripts/ # Install/uninstall scripts -│ └── systemd/ # Systemd service files -└── docs/ # Documentation -``` - -## Building Packages - -Packages are built using [GoReleaser](https://goreleaser.com), which handles .deb, .rpm, and tarball creation automatically. - -### Install GoReleaser - -```bash -# Using Homebrew (macOS/Linux) -brew install goreleaser - -# Using snap (Linux) -snap install --classic goreleaser - -# Or download from https://github.com/goreleaser/goreleaser/releases -``` - -### Build Snapshot Packages - -Build packages without a git tag (for testing): - -```bash -# Using Make -make package-snapshot - -# Or directly with GoReleaser -goreleaser release --snapshot --clean -``` - -This creates in the `dist/` directory: -- `.deb` packages for Debian/Ubuntu (amd64, arm64) -- `.rpm` packages for RHEL/Rocky/AlmaLinux 8, 9, 10 (x86_64, aarch64) -- Binary tarballs for direct installation -- Checksums and metadata - -### Create a Release - -To create an official release, tag a version and push it to GitHub. The GitHub Actions workflow will automatically build and publish the release: - -```bash -# Tag a version -git tag v0.2.0 - -# Push the tag to trigger the release workflow -git push origin v0.2.0 -``` - -The GitHub Actions workflow will: -- Build packages for all supported platforms (amd64, arm64) -- Create `.deb` packages for Ubuntu -- Create `.rpm` packages for RHEL/Rocky/AlmaLinux 8, 9, 10 -- Generate binary tarballs -- Publish the release with all artifacts to GitHub Releases - -## Testing - -### Run Tests with Coverage - -```bash -make test -``` - -This runs all tests and generates a coverage report in `coverage/coverage.html`. - -### Run Linting - -```bash -make lint -``` - -Runs `golangci-lint` if installed, otherwise runs basic Go formatting and vet checks. - -### Format Code - -```bash -make fmt -``` - -### Vulnerability Scanning - -```bash -make vuln -``` - -Scans the built binary for known vulnerabilities using `govulncheck`. - -### Manual Testing - -```bash -# Build and run locally -make fleetint -./bin/fleetint scan - -# Test with different configurations -./bin/fleetint run --log-level=debug --listen-address=127.0.0.1:8080 - -# Test offline mode -./bin/fleetint run --offline-mode --path=/tmp/test --duration=00:01:00 --format=csv -``` - -## Development Workflow - -1. **Create a branch** for your changes -2. **Make changes** and add tests -3. **Run tests**: `make test` -4. **Format code**: `make fmt` -5. **Run linting**: `make lint` -6. **Build locally**: `make fleetint` -7. **Test changes** manually -8. **Build packages** (optional): `make package-snapshot` -9. **Submit PR** with description of changes - -## Code Guidelines - -- Follow standard Go conventions and idioms -- Add tests for new functionality -- Update documentation for user-facing changes -- Ensure all tests pass before submitting PRs -- Use `make fmt` to format code before committing -- Keep commits focused and write clear commit messages - -## Available Make Targets - -```bash -make help # Show all available targets -make all # Build all binaries -make fleetint # Build fleetint binary -make test # Run tests with coverage -make lint # Run linting tools -make fmt # Format Go code -make vuln # Run vulnerability check -make clean # Clean up binaries and artifacts -make package-snapshot # Build packages (no git tag required) -``` - -## Related Projects - -- **[leptonai/gpud](https://github.com/leptonai/gpud)**: Upstream dependency providing core monitoring functionality - -## Contributing - -See [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed contribution guidelines. diff --git a/docs/install-deb.md b/docs/install-deb.md deleted file mode 100644 index 6ea3e246..00000000 --- a/docs/install-deb.md +++ /dev/null @@ -1,90 +0,0 @@ -# DEB Installation - -## Prerequisites - -Before installing Fleet Intelligence Agent, ensure the following prerequisites are met: - -- Configure an NVIDIA package repository for automatic dependency installation. - The following dependencies are required and satisfied by the CUDA (network or local) repository. - - `datacenter-gpu-manager-4-proprietary` (DCGM) - - `nvattest` (NVIDIA Attestation SDK CLI, NVAT) - - `corelib` (NVAT GPU evidence source dependency) -- DCGM HostEngine `4.2.3` or newer -- NVIDIA Datacenter Driver major version `510` or newer is installed -- Install/upgrade commands are run as `root` or with `sudo` -- Attestation for the fleetint use case only supports Blackwell and newer GPUs, and applies to non-CC mode systems - - Fleetint is supported without attestation for Hopper GPUs. -- For NVSwitch systems (driver branch must match installed datacenter driver): - - Hopper (pre-4th gen NVSwitch): install `cuda-drivers-fabricmanager-` - - Blackwell (4th gen NVSwitch): install `nvidia-open-` and `nvlink5-` - -Fleet Intelligence Agent package dependencies are available through NVIDIA's CUDA repository. Before installing Fleet Intelligence Agent, add the appropriate NVIDIA CUDA repository for your system: - -```bash -# Ubuntu 22.04 (x86_64) -wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb -sudo dpkg -i cuda-keyring_1.1-1_all.deb -sudo apt-get update - -# Ubuntu 24.04 (x86_64) -wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb -sudo dpkg -i cuda-keyring_1.1-1_all.deb -sudo apt-get update - -# Ubuntu 22.04 (ARM64) -wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/cuda-keyring_1.1-1_all.deb -sudo dpkg -i cuda-keyring_1.1-1_all.deb -sudo apt-get update - -# Ubuntu 24.04 (ARM64) -wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb -sudo dpkg -i cuda-keyring_1.1-1_all.deb -sudo apt-get update -``` - -After adding the CUDA repository, package dependencies (`datacenter-gpu-manager-4-proprietary`, `nvattest`, and `corelib`) are resolved automatically by `apt` during Fleet Intelligence Agent installation. - -## Install package - -Download the package from [Latest stable release](https://github.com/NVIDIA/fleet-intelligence-agent/releases/latest), then install: - -```bash -# Ubuntu (x86_64) -sudo apt install ./fleetint_VERSION_amd64.deb - -# Ubuntu (ARM64) -sudo apt install ./fleetint_VERSION_arm64.deb -``` - -Verify: - -```bash -sudo fleetint --version -systemctl status fleetintd -``` - -## Update - -Install the newer package version: - -```bash -# Ubuntu (x86_64) -sudo apt install ./fleetint_VERSION_amd64.deb - -# Ubuntu (ARM64) -sudo apt install ./fleetint_VERSION_arm64.deb -``` - -The service will automatically restart with the new version. - -## Uninstall - -```bash -sudo apt remove fleetint -sudo apt purge fleetint # Also removes configuration files -``` - -References: -- DCGM: -- Fabric Manager: -- NVAT (`nvattest`/`corelib`): \ No newline at end of file diff --git a/docs/install-helm.md b/docs/install-helm.md deleted file mode 100644 index c7aac70b..00000000 --- a/docs/install-helm.md +++ /dev/null @@ -1,245 +0,0 @@ -# Helm Installation - -## Prerequisites - -- NVIDIA GPU Operator installed with DCGM HostEngine enabled. -- NVIDIA Datacenter Driver major version `510` or newer on the cluster nodes. -- DCGM HostEngine `4.2.3` or newer. -- A DCGM service endpoint reachable from the cluster (defaults to `nvidia-dcgm.gpu-operator.svc:5555`). -- Access to GitHub Container Registry (`ghcr.io`) from your cluster/network. - -Set shared variables once for the examples below: - -```bash -# Namespace (override if needed) -NS=fleet-intelligence - -CHART_VERSION='' # e.g. 0.3.2 or 0.3.2-rc.1 - -# DCGM endpoint (usually the default is correct) -DCGM_URL='nvidia-dcgm.gpu-operator.svc:5555' - -# Enrollment configuration - Go to the Fleet Intelligence UI to: -# 1. Generate an enrollment token (ENROLL_TOKEN) -# 2. Get the enrollment endpoint URL (ENROLL_ENDPOINT) -ENROLL_ENDPOINT='' -ENROLL_TOKEN='' -ENROLL_TOKEN_SECRET_NAME='fleet-intelligence-enroll-token' # Recommended secret name -``` - -## Create namespace - -```bash -kubectl create namespace "$NS" || true -``` - -## Create enrollment secret - -If you need to enroll nodes, create the token Secret. The secret name should match the `ENROLL_TOKEN_SECRET_NAME` variable set above: - -```bash -kubectl create secret generic "$ENROLL_TOKEN_SECRET_NAME" \ - --namespace "$NS" \ - --from-literal=token="$ENROLL_TOKEN" -``` - -## Install or upgrade - -Install: - -```bash -helm install fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --set enroll.enabled=true \ - --set enroll.endpoint="$ENROLL_ENDPOINT" \ - --set enroll.tokenSecretName="$ENROLL_TOKEN_SECRET_NAME" -``` - -Install (no enrollment): - -```bash -helm install fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" -``` - -Upgrade: - -```bash -helm upgrade fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --set enroll.enabled=true \ - --set enroll.endpoint="$ENROLL_ENDPOINT" \ - --set enroll.tokenSecretName="$ENROLL_TOKEN_SECRET_NAME" -``` - -Optional: include node metadata during automatic enrollment: - -```bash -helm upgrade fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --set enroll.enabled=true \ - --set enroll.endpoint="$ENROLL_ENDPOINT" \ - --set enroll.tokenSecretName="$ENROLL_TOKEN_SECRET_NAME" \ - --set-string enroll.nodeGroup="prod-a" \ - --set-string enroll.computeZone="us-east-1c" -``` - -Notes: -- Omit `enroll.nodeGroup` / `enroll.computeZone` keys to omit the flags and preserve existing stored values. -- Set either value to an empty string to clear it (for example: `--set-string enroll.nodeGroup=""`). - -Upgrade (no enrollment): - -```bash -helm upgrade fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" -``` - -Upgrade and explicitly remove persisted enrollment metadata: - -```bash -helm upgrade fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --set enroll.enabled=false \ - --set enroll.unenroll=true -``` - -`enroll.enabled` and `enroll.unenroll` are mutually exclusive. Setting both to `true` causes Helm template rendering to fail. - -To use a different image registry/repository, add: - -```bash ---set image.repository="" -``` - -If DCGM is exposed at a different service name or port, set `env.DCGM_URL`: - -```bash ---set env.DCGM_URL="$DCGM_URL" -``` - -## Verifying deployment - -After installation, verify the agent is running correctly: - -```bash -# Check DaemonSet status -kubectl get daemonset fleet-intelligence-agent -n "$NS" - -# Check pods (should be one per GPU node) -kubectl get pods -n "$NS" -l app.kubernetes.io/name=fleet-intelligence-agent - -# View pod logs -kubectl logs -n "$NS" -l app.kubernetes.io/name=fleet-intelligence-agent --tail=50 - -# Watch rollout status -kubectl rollout status daemonset/fleet-intelligence-agent -n "$NS" -``` - -Check a specific pod in detail: - -```bash -# Get a pod name -POD_NAME=$(kubectl get pods -n "$NS" -l app.kubernetes.io/name=fleet-intelligence-agent -o jsonpath='{.items[0].metadata.name}') - -# Describe the pod -kubectl describe pod -n "$NS" "$POD_NAME" - -# View full logs -kubectl logs -n "$NS" "$POD_NAME" --follow -``` - -## Troubleshooting - -**Pods not starting:** - -```bash -# Check pod events -kubectl describe pod -n "$NS" -l app.kubernetes.io/name=fleet-intelligence-agent -``` - -Common issues: -- **ImagePullBackOff**: Verify nodes can reach `ghcr.io` and the image tag exists -- **Pending**: Check node labels match `nodeSelector` (default: `nvidia.com/gpu.deploy.dcgm=true`) -- **CrashLoopBackOff**: Check logs for errors - -**Enrollment failures:** - -```bash -# Check init container logs -kubectl logs -n "$NS" "$POD_NAME" -c enroll - -# Verify enrollment secret exists -kubectl get secret "$ENROLL_TOKEN_SECRET_NAME" -n "$NS" - -# Check secret content (verify token is not empty) -kubectl get secret "$ENROLL_TOKEN_SECRET_NAME" -n "$NS" -o jsonpath='{.data.token}' | base64 -d | wc -c -``` - -**DCGM connection issues:** - -```bash -# Verify DCGM service is accessible -kubectl get svc -n gpu-operator nvidia-dcgm - -# Test DCGM connectivity from a pod -kubectl exec -n "$NS" "$POD_NAME" -- curl -v telnet://nvidia-dcgm.gpu-operator.svc:5555 - -# Check DCGM URL environment variable -kubectl get pods -n "$NS" "$POD_NAME" -o jsonpath='{.spec.containers[0].env[?(@.name=="DCGM_URL")].value}' -``` - -If DCGM is at a different location, update the URL: - -```bash -helm upgrade fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --reuse-values \ - --set env.DCGM_URL=":" -``` - -## Node Scheduling - -**By default**, the agent only deploys to nodes where DCGM is running, using the nodeSelector: - -```yaml -nodeSelector: - nvidia.com/gpu.deploy.dcgm: "true" -``` - -The agent requires a DCGM HostEngine to collect GPU metrics, so it must co-locate with DCGM. This label is -automatically set by the NVIDIA GPU Operator when DCGM is enabled — no manual labeling is required. - -If you need a different node selector or tolerations for GPU taints, you can override them. -The examples below use a generic label to illustrate the override syntax — replace it with the actual label used in your cluster. - -Using `--set` (quote the tolerations for zsh, and escape dots in the label key): - -```bash -helm upgrade --install fleet-intelligence-agent oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \ - --version "$CHART_VERSION" \ - --namespace "$NS" \ - --set-string nodeSelector.my-org\\.com/gpu-node=true \ - --set 'tolerations[0].key=nvidia.com/gpu' \ - --set 'tolerations[0].operator=Exists' \ - --set 'tolerations[0].effect=NoSchedule' -``` - -Using a values file: - -```yaml -nodeSelector: - my-org.com/gpu-node: "true" - -tolerations: - - key: "nvidia.com/gpu" - operator: "Exists" - effect: "NoSchedule" -``` diff --git a/docs/install-rpm.md b/docs/install-rpm.md deleted file mode 100644 index ce32f4ff..00000000 --- a/docs/install-rpm.md +++ /dev/null @@ -1,114 +0,0 @@ -# RPM Installation - -## Prerequisites - -Before installing Fleet Intelligence Agent, ensure the following prerequisites are met: - -- Configure an NVIDIA package repository for automatic dependency installation. - The following dependencies are required and satisfied by the CUDA (network or local) repository. - - `datacenter-gpu-manager-4-proprietary` (DCGM) - - `nvattest` (NVIDIA Attestation SDK CLI, NVAT) - - `corelib` (NVAT GPU evidence source dependency) -- DCGM HostEngine `4.2.3` or newer -- NVIDIA Datacenter Driver major version `510` or newer is installed -- Install/upgrade commands are run as `root` or with `sudo` -- `dnf-plugins-core` is installed (required for `dnf config-manager`) -- Attestation for the fleetint use case only supports Blackwell and newer GPUs, and applies to non-CC mode systems - - Fleetint is supported without attestation for Hopper GPUs. -- For NVSwitch systems (driver branch must match installed datacenter driver): - - Hopper (pre-4th gen NVSwitch): install `nvidia-driver:/fm` - - Blackwell (4th gen NVSwitch): install `nvidia-driver--open` and `nvlink-` - -Fleet Intelligence Agent package dependencies are available through NVIDIA's CUDA repository. Before installing Fleet Intelligence Agent, add the appropriate NVIDIA CUDA repository for your system: - -Install `dnf-plugins-core` first if `dnf config-manager` is not available: - -```bash -sudo dnf install -y dnf-plugins-core -``` - -### RHEL/Rocky/AlmaLinux Systems - -```bash -# RHEL/Rocky/AlmaLinux 8 (x86_64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo -sudo dnf clean all - -# RHEL/Rocky/AlmaLinux 9 (x86_64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo -sudo dnf clean all - -# RHEL/Rocky/AlmaLinux 10 (x86_64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel10/x86_64/cuda-rhel10.repo -sudo dnf clean all - -# RHEL/Rocky/AlmaLinux 8 (ARM64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/sbsa/cuda-rhel8.repo -sudo dnf clean all - -# RHEL/Rocky/AlmaLinux 9 (ARM64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/sbsa/cuda-rhel9.repo -sudo dnf clean all - -# RHEL/Rocky/AlmaLinux 10 (ARM64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel10/sbsa/cuda-rhel10.repo -sudo dnf clean all -``` - -### Amazon Linux 2023 - -```bash -# Amazon Linux 2023 (x86_64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/x86_64/cuda-amzn2023.repo -sudo dnf clean all - -# Amazon Linux 2023 (ARM64) -sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/amzn2023/sbsa/cuda-amzn2023.repo -sudo dnf clean all -``` - -After adding the CUDA repository, package dependencies (`datacenter-gpu-manager-4-proprietary`, `nvattest`, and `corelib`) are resolved automatically by `dnf` during Fleet Intelligence Agent installation. - -## Install package - -Download the package from [Latest stable release](https://github.com/NVIDIA/fleet-intelligence-agent/releases/latest), then install: - -```bash -# RHEL/Rocky/AlmaLinux/Amazon Linux (x86_64) -sudo dnf install ./fleetint-VERSION-1.x86_64.rpm - -# RHEL/Rocky/AlmaLinux/Amazon Linux (ARM64) -sudo dnf install ./fleetint-VERSION-1.aarch64.rpm -``` - -Verify: - -```bash -sudo fleetint --version -systemctl status fleetintd -``` - -## Update - -Install the newer package version: - -```bash -# RHEL/Rocky/AlmaLinux/Amazon Linux (x86_64) -sudo dnf install ./fleetint-VERSION-1.x86_64.rpm - -# RHEL/Rocky/AlmaLinux/Amazon Linux (ARM64) -sudo dnf install ./fleetint-VERSION-1.aarch64.rpm -``` - -The service will automatically restart with the new version. - -## Uninstall - -```bash -sudo dnf remove fleetint -``` - -References: -- DCGM: -- Fabric Manager: -- NVAT (`nvattest`/`corelib`): \ No newline at end of file diff --git a/docs/usage.md b/docs/usage.md deleted file mode 100644 index 630e3b8e..00000000 --- a/docs/usage.md +++ /dev/null @@ -1,392 +0,0 @@ -# Usage - -## Basic Commands - -### Quick Health Scan - -```bash -sudo fleetint scan -``` - -Performs a quick health scan of GPUs and system components. Returns immediately with a summary of any detected issues. - -**Aliases:** `sudo fleetint check`, `sudo fleetint s` - -### Start Monitoring Server - -```bash -sudo fleetint run -``` - -Starts the API server. By default it listens on a Unix socket at `/run/fleetint/fleetint.sock` (access controlled by file permissions). Pass `--listen-address` to switch to TCP. - -**Options:** -- `--log-level`: Set logging level (debug, info, warn, error) -- `--listen-address`: Listen address. An absolute path (e.g. `/run/fleetint/fleetint.sock`) creates a Unix socket; a `host:port` value (e.g. `127.0.0.1:15133`) opens a TCP listener. Default: `/run/fleetint/fleetint.sock`. See [Exposing the Agent for External Monitoring](#exposing-the-agent-for-external-monitoring) for details on exposing to Prometheus and other tools. -- `--components`: Enable/disable specific components - -### Check Status - -```bash -sudo fleetint status -``` - -Displays the current status of the fleetint service and monitored components. - -**Alias:** `sudo fleetint st` - -### Machine Information - -```bash -sudo fleetint machine-info -``` - -Shows detailed information about the machine: -- Hardware specifications (CPU, memory, disk) -- GPU configuration and driver version -- CUDA runtime version -- Operating system and kernel version -- Network interfaces -- System UUID and machine ID - -### View/Update Metadata - -```bash -# View current metadata -sudo fleetint metadata - -# Set metadata key-value pair -sudo fleetint metadata --set-key="key" --set-value="value" -``` - -Used to view or update the agent's metadata store, including remote export configuration. - -### Compact State Database - -```bash -sudo fleetint compact -``` - -Compacts the local Fleet Intelligence state database to reduce disk usage. - -Requirements: -- `fleetintd` must be stopped before running `compact` -- the agent must not be running (no active listener on the default socket or port) -- the command needs write access to the state database, so package installs typically require `sudo` - -Typical workflow: - -```bash -sudo systemctl stop fleetintd -sudo fleetint compact -sudo systemctl start fleetintd -``` - -### Validate Prerequisites - -```bash -sudo fleetint precheck -``` - -Validates the local prerequisites required for installation and enrollment. - -**What it checks:** -- NVIDIA GPU presence -- supported GPU architecture (`Ampere`, `Ada Lovelace`, `Hopper`, `Blackwell`, `Rubin`) -- NVIDIA driver major version (`510` or newer) -- DCGM HostEngine reachability -- DCGM HostEngine minimum version (`4.2.3`) -- `nvattest` availability - -The command prints each check result and exits non-zero if any hard requirement fails. - -**Environment Variables (DCGM connection):** -- `DCGM_URL`: Address of the DCGM HostEngine (default: `localhost`) -- `DCGM_URL_IS_UNIX_SOCKET`: Set to `true` if `DCGM_URL` is a Unix socket path (default: `false`) - -### Enroll Agent - -```bash -# Pass token directly (visible in process list) -sudo fleetint enroll --endpoint=https://api.example.com --token= - -# Read token from a file (recommended) -sudo fleetint enroll --endpoint=https://api.example.com --token-file=/path/to/token - -# Read token from stdin -echo "$TOKEN" | sudo fleetint enroll --endpoint=https://api.example.com --token-file=- -``` - -Enrolls the agent with the Fleet Intelligence backend by exchanging a Service Account Key (SAK) token for a JWT token. The JWT token and backend endpoints are stored locally for subsequent data exports. - -**Required Options:** -- `--endpoint`: Base endpoint URL for the Fleet Intelligence backend (must use HTTPS) -- `--token`: Service Account Key (SAK) token for authentication (mutually exclusive with `--token-file`) -- `--token-file`: Path to a file containing the SAK token, or `-` to read from stdin (mutually exclusive with `--token`). Preferred over `--token` because it avoids exposing the token in `/proc//cmdline`. - -One of `--token` or `--token-file` is required. - -**Optional Flags:** -- `--force`: Continue enrollment even if `fleetint precheck` fails -- `--node-group`: Optional node group metadata persisted in local agent metadata -- `--compute-zone`: Optional compute zone metadata persisted in local agent metadata - -Metadata update behavior for `--node-group` and `--compute-zone`: -- If the flag is omitted, the existing stored value is preserved. -- If the flag is provided, the stored value is overwritten with the provided value. -- Providing an empty value (for example `--node-group=""`) clears the stored value. - -**What it does:** -1. Runs the same prerequisite validation as `fleetint precheck` -2. Validates the endpoint URL (must be HTTPS) -3. Makes an enrollment request to exchange the SAK token for a JWT token -4. Stores the JWT token, backend endpoints (metrics, logs, nonce), and optional enrollment metadata (`node_group`, `compute_zone`) in the local metadata database -5. The stored credentials are used automatically by the agent for data export - -**Example output:** -``` -Enrollment succeeded -``` - -**Error handling:** -- precheck failure: Enrollment is blocked unless `--force` is set -- 400: Token format is incorrect -- 401: Token is invalid -- 403: Token is expired or revoked -- 404: Endpoint not found -- 429: Server is rate limiting (retry later) -- 502/503/504: Temporary server issues (retry) - -### Unenroll Agent - -```bash -sudo fleetint unenroll -``` - -Removes all enrollment credentials and backend endpoints from the agent. After unenrolling, the agent will no longer export data to the backend until re-enrolled. - -**What it does:** -1. Clears the JWT token from local storage -2. Clears the SAK token from local storage -3. Removes all stored backend endpoints (enroll, metrics, logs, nonce) - -Use this command when: -- Decommissioning a machine -- Switching to a different backend -- Troubleshooting authentication issues - -## Offline Data Collection - -For environments without network access or when you need to collect data to files: - -```bash -sudo fleetint run --offline-mode --path=/tmp/fleetint --duration=00:05:00 --format=csv -``` - -**Options:** -- `--offline-mode`: Disable HTTP API server and export to files -- `--path`: Absolute path to the output directory. Must not be inside restricted system directories (`/etc`, `/usr`, `/sys`, `/bin`, `/boot`, `/dev`, `/lib`, `/proc`, `/run`, `/sbin`). -- `--duration`: How long to collect data (format: HH:MM:SS) -- `--format`: Output format (`csv` or `json`) - -## Running as a Service - -After package installation, the agent runs as a systemd service: - -```bash -# Check service status -sudo systemctl status fleetintd - -# Start/stop/restart service -sudo systemctl start fleetintd -sudo systemctl stop fleetintd -sudo systemctl restart fleetintd - -# View logs -sudo journalctl -u fleetintd -f -``` - -## HTTP API - -The fleetint API server listens on a Unix socket (`/run/fleetint/fleetint.sock`) by default. When started with a TCP address (e.g. `--listen-address=127.0.0.1:15133`), the REST endpoints are also available over plain HTTP. - -**Using curl with the default Unix socket** (requires sudo since the socket is owner-only): - -```bash -sudo curl --unix-socket /run/fleetint/fleetint.sock http://localhost/healthz -``` - -**Using curl with TCP** (requires `--listen-address=127.0.0.1:15133`): - -```bash -curl http://localhost:15133/healthz -``` - -The examples below use the TCP form for brevity. For the default socket, prefix with `sudo` and substitute `--unix-socket /run/fleetint/fleetint.sock http://localhost` for the hostname. - -### Health Check - -```bash -curl http://localhost:15133/healthz -``` - -Returns the health status of the API server - -**Response:** -```json -{ - "status": "ok", - "version": "v1" -} -``` - -### Machine Information - -```bash -curl http://localhost:15133/machine-info -``` - -Returns basic machine info - -Note: Detailed hardware and GPU information is available via the `fleetint machine-info` CLI command. - -### Current Health States - -```bash -curl http://localhost:15133/v1/states -``` - -Returns the current health states of all monitored components - -### Component Metrics - -```bash -curl http://localhost:15133/v1/metrics -``` - -Returns metrics data in JSON format from all monitored components - -**Query Parameters:** -- `startTime`: Unix timestamp to retrieve metrics since a specific time -- `components`: Filter metrics by component name - -**Example:** -```bash -# Get metrics from the last hour -curl "http://localhost:15133/v1/metrics?startTime=$(date -d '1 hour ago' +%s)" - -# Get metrics for specific component -curl "http://localhost:15133/v1/metrics?components=accelerator-nvidia-temperature" -``` - -### Component Events - -```bash -curl http://localhost:15133/v1/events -``` - -Returns event data from all monitored components (errors, warnings, state changes) - -**Query Parameters:** -- `since`: Unix timestamp to retrieve events since a specific time (default: last hour) -- `components`: Filter events by component name - -**Example:** -```bash -# Get events from the last hour -curl "http://localhost:15133/v1/events?since=$(date -d '1 hour ago' +%s)" - -# Get events for specific component -curl "http://localhost:15133/v1/events?components=accelerator-nvidia-error-xid" -``` - -### Prometheus Metrics - -```bash -curl http://localhost:15133/metrics -``` - -Returns metrics in Prometheus exposition format for integration with monitoring systems - -## Exposing the Agent for External Monitoring - -By default, fleetint uses a Unix socket for security. To allow external monitoring tools like Prometheus to scrape metrics over the network, switch to a TCP listener with the `--listen-address` flag: - -```bash -# Expose on all interfaces -sudo fleetint run --listen-address=0.0.0.0:15133 - -# Or expose on a specific IP address -sudo fleetint run --listen-address=192.168.1.100:15133 -``` - -### Prometheus Configuration Example - -Configure Prometheus to scrape the exposed endpoint: - -```yaml -# prometheus.yml -scrape_configs: - - job_name: 'fleetint' - scrape_interval: 60s - static_configs: - - targets: - - 'gpu-server-1:15133' - - 'gpu-server-2:15133' - metrics_path: /metrics -``` - -**For production deployments** with persistent configuration and security considerations, see the [Configuration Guide](configuration.md#where-to-configure). - -## Troubleshooting - -### Service won't start - -1. Check service status: - ```bash - sudo systemctl status fleetintd - ``` - -2. View recent logs: - ```bash - sudo journalctl -u fleetintd -n 50 - ``` - -3. Verify NVIDIA drivers are installed (if using GPUs): - ```bash - nvidia-smi - ``` - -4. Check that the daemon is listening: - ```bash - # Default (unix socket) - sudo ls -la /run/fleetint/fleetint.sock - - # TCP mode - sudo netstat -tlnp | grep 15133 - ``` - -### Export issues - -1. Check the logs: - ```bash - sudo journalctl -u fleetintd -f - ``` - -2. Verify your configuration: - ```bash - sudo fleetint metadata - ``` - -3. Test connectivity to the export endpoint manually with `curl` - -4. Check proxy settings in `/etc/default/fleetint` if behind a firewall - -### High resource usage - -The agent should use <500MB RAM and <1% CPU. Higher usage might indicate: - -- Very frequent collection intervals (check `FLEETINT_COLLECT_INTERVAL`) -- Large lookback windows (check `FLEETINT_METRICS_LOOKBACK` and `FLEETINT_EVENTS_LOOKBACK`) -- Many GPUs in the system (resource usage scales with GPU count) -- Debug logging enabled (use `--log-level=warn` or `error`)