Skip to content

feat: add authenticated OpenTelemetry gateway #231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rsampaio wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add authenticated OpenTelemetry gateway #231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
.git
.github
.cache
coverage
dist
bin
build
otelcol/bin
**/*.test
**/*.out
**/*.log
Expand Down
34 changes: 34 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,40 @@ jobs:
path: fleetint-${{ matrix.goos }}-${{ matrix.goarch }}
retention-days: 7

build_check_otelcol:

@ambermingxin ambermingxin Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

otelcol/auth/sakauth is a separate Go module (it has its own go.mod). In .github/workflows/ci.yml, unit_test runs make test → go test ./... from the root module, which does not descend into nested modules. similar to the lint runs golangci-lint at the root and vuln runs govulncheck ./... at the root. build_check_otelcol only runs the single integration test: run ^TestCollectorGatewayEndToEnd$ and only for amd64). so extension_test.go and config tests are never executed in CI. Should we add something like following in ci to include that?

- name: Test sakauth module
  working-directory: otelcol/auth/sakauth
  run: go test -race ./...

name: Build Check (otelcol)
runs-on: ubuntu-latest
needs: [lint]
strategy:
matrix:
goarch: [amd64, arm64]
steps:
- name: Checkout code
uses: actions/checkout@v4
Comment on lines +188 to +189

@coderabbitai coderabbitai Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Disable credential persistence on checkout for this build job.

This job installs and runs third-party build tooling (go install .../builder@v0.156.0, then compiles further transitive dependencies via ocb). Without persist-credentials: false, the GITHUB_TOKEN is persisted in git config and readable by any code executed later in the job, widening the blast radius if a fetched dependency is compromised.

🔒 Proposed fix 
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Checkout code
uses: actions/checkout@v4
- name: Checkout code
uses: actions/checkout@v4
with:
persist-credentials: false
🧰 Tools
🪛 zizmor (1.26.1)

[warning] 188-189: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 188 - 189, Update the
actions/checkout@v4 step in the build job to set persist-credentials to false,
ensuring the checkout action does not store the GITHUB_TOKEN in git
configuration before third-party tooling runs.

Source: Linters/SAST tools


- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: ${{ env.GO_VERSION }}

- name: Install OTel Collector Builder
run: go install go.opentelemetry.io/collector/cmd/builder@v0.156.0
Comment on lines +191 to +197

@coderabbitai coderabbitai Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the workflow trigger and relevant job section.
file=".github/workflows/ci.yml"
echo "== File lines around trigger =="
nl -ba "$file" | sed -n '1,120p'
echo
echo "== File lines around setup-go step =="
nl -ba "$file" | sed -n '170,220p'

Repository: NVIDIA/fleet-intelligence-agent

Length of output: 241

🌐 Web query:

actions/setup-go v6 cache default enabled cache behavior workflow cache poisoning GitHub Actions

💡 Result:

The actions/setup-go action (including v6) has caching enabled by default [1][2]. The action supports caching for Go modules and build outputs to improve performance, using the cache input which defaults to true [1][3]. Regarding cache poisoning, this is a known security concern in GitHub Actions environments [4][5]. Because actions/setup-go historically enables read-write caching by default, untrusted workflows—such as those triggered by pull requests from forks or external contributors—can potentially write malicious artifacts to the shared cache [4][6]. These tainted artifacts can then be restored by subsequent, trusted workflow runs on the main branch, leading to a supply chain vulnerability [4][5]. To mitigate this risk, several strategies are employed: 1. Manual Disabling: In sensitive workflows, such as release or deployment pipelines, it is standard practice to explicitly set cache: false to prevent the action from writing to the cache [5][6]. 2. Read-Only Cache Support: Recent updates to actions/setup-go include the addition of a cache-write input [4]. By setting cache-write: false, users can allow the action to restore existing caches while skipping the post-step cache save, effectively implementing a read-only cache mode [4]. This allows workflows (like PR checks) to benefit from existing dependencies without risking the integrity of the cache for future runs [4]. When designing workflows, it is recommended to evaluate whether full read-write caching is necessary or if a read-only configuration provides sufficient protection against cache poisoning [4][5].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file=".github/workflows/ci.yml"

echo "== Trigger and top-level workflow lines =="
sed -n '1,90p' "$file" | cat -n

echo
echo "== Step around actions/setup-go =="
sed -n '180,205p' "$file" | cat -n

Repository: NVIDIA/fleet-intelligence-agent

Length of output: 4316

🌐 Web query:

actions/setup-go v4 deprecated Node runtime current major v6 release notes

💡 Result:

The actions/setup-go action has progressed through several major versions, primarily driven by the deprecation and end-of-life cycles of Node.js runtimes on GitHub Actions runners [1][2][3]. Key points regarding the transition and v6 release: 1. Node.js Runtime Updates: Older versions of the action used outdated Node.js runtimes. Specifically, v5 upgraded the runtime from Node 16 to Node 20 [2]. The current major release, v6, further upgrades the Node.js runtime from Node 20 to Node 24 [3]. 2. Deprecation Context: The deprecation of Node 20 on GitHub Actions runners was announced in September 2025, as Node 20 reached its end-of-life (EOL) in April 2026 [1]. Users are encouraged to update to actions that support newer Node.js runtimes, such as v6 of setup-go [4][3]. 3. v6 Release Notes: The v6.0.0 release (published September 4, 2025) includes the upgrade to Node 24 and improvements to toolchain handling [3]. Because of the runtime change, users must ensure their runners are on version v2.327.1 or later to maintain compatibility [4][3]. Additional enhancements in the v6.x series have included updated download sources for Go distributions and support for.tool-versions files [5]. In summary, the transition from v4 to v6 involves significant updates to the underlying Node.js runtime (from Node 16 in v4/early versions to Node 24 in v6) to maintain compliance with GitHub Actions runner lifecycle requirements [2][3].

Citations:

🌐 Web query:

GitHub Actions pull_request fork cache save actions/setup-go default cache poisoning trusted push restore

💡 Result:

When using GitHub Actions, specifically with pull_request events from forks or pull_request_target workflows, understanding cache access and security risks is critical. Cache Access Limitations By default, GitHub Actions enforces cache isolation [1][2]. A workflow run triggered by a pull request from a fork has restricted access to the cache: - It can restore existing cache entries available to the base branch or default branch [1]. - It is generally restricted from creating or overwriting cache entries to prevent unauthorized cache modifications [3][1]. When a cache write is denied in these scenarios, the actions/cache step reports a warning but does not fail the job [3][4]. Security Risks: Cache Poisoning "Cache poisoning" is a security concern where an attacker attempts to inject malicious data into the cache [5][6][7]. - Risk with pull_request_target: This trigger runs in the context of the base repository and has access to secrets and write permissions [8][4][6]. If a workflow using pull_request_target is misconfigured to check out and execute code from an untrusted fork (the "Pwn Request" pattern), that code runs with the privileges of the base repository [5][6][7]. - Mechanism: Even if you restrict the job's GITHUB_TOKEN permissions (e.g., contents: read), the cache save step uses an internal, runner-specific token to authenticate with the cache service [5][6][7]. Consequently, if malicious code runs during the workflow, it can write arbitrary data to the cache under a deterministic key that a future, trusted workflow might restore [6][7]. This can lead to downstream supply-chain compromises [5][7]. Best Practices for actions/setup-go and Caching The actions/setup-go action uses actions/cache internally [9][10]. To maintain security: - Avoid Unsafe Saves: Do not allow cache writes in workflows that execute untrusted fork code (such as those using pull_request_target that check out the PR's head ref) [5][11]. - Use Restore-Only Logic: In untrusted contexts, configure workflows to be "restore-only" [5]. You can implement custom logic to detect if a PR is from a fork and modify your cache keys to prevent collision with trusted branches [12]. - Rely on Trusted Triggers: Only perform cache saves in workflows triggered by trusted events, such as push to your default branch [8][4]. For further implementation guidance, refer to the documentation on securing pull_request_target workflows and the specific behavior of the actions/cache action [8][3][4].

Citations:

🌐 Web query:

GitHub Actions cache forked pull requests read-only save disabled official docs

💡 Result:

GitHub Actions restricts cache access for workflows triggered by pull requests from forks to ensure security and prevent cache poisoning [1][2]. In these scenarios, workflows have read-only access to the cache, meaning they can restore existing cache entries but cannot save or create new ones [1][3]. When a workflow attempts to save a cache while in a read-only state: - The job does not fail [1][3]. - The actions/cache action reports a warning (e.g., "cache write denied") in the workflow logs [1][3]. - The step completes successfully without writing any data to the cache [1][3]. This behavior is a security measure applied to untrusted triggers, including pull requests from forks, to prevent unauthorized modification of the shared cache [4][2]. For workflows that require populating the cache, GitHub recommends using trusted triggers, such as events on the base repository's default branch (e.g., push events) [2]. As of June 2026, GitHub has further reinforced read-only cache protections for workflow events that can be triggered without explicit write permissions [4].

Citations:

Update actions/setup-go to v6. actions/setup-go@v4 is behind the current major and should be bumped to avoid Node-runtime deprecation issues.

🧰 Tools
🪛 actionlint (1.7.12)

[error] 192-192: the runner of "actions/setup-go@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue

(action)

🪛 zizmor (1.26.1)

[error] 192-192: runtime artifacts potentially vulnerable to a cache poisoning attack (cache-poisoning): enables caching by default

(cache-poisoning)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 191 - 197, Update the “Set up Go”
workflow step to use actions/setup-go@v6 instead of v4, keeping its existing
go-version configuration unchanged.

Source: Linters/SAST tools


- name: Build fleetint-otelcol
working-directory: otelcol
env:
GOOS: linux
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 0
run: builder --config=otelcol-builder.yaml

- name: Run OTel gateway integration test
if: matrix.goarch == 'amd64'
working-directory: otelcol/auth/sakauth
env:
FLEETINT_OTELCOL_INTEGRATION: "1"
run: go test -race -run '^TestCollectorGatewayEndToEnd$' .

codeql:
name: CodeQL Analysis
runs-on: ubuntu-latest
Expand Down
27 changes: 27 additions & 0 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,21 @@ jobs:
fi
echo "EOF" >> "$GITHUB_OUTPUT"

- name: Prepare fleetint-otelcol image tags
id: otelcol_image
run: |
set -euo pipefail
VERSION="${{ steps.version.outputs.version }}"
OWNER_LC="$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')"
GHCR_IMAGE="ghcr.io/${OWNER_LC}/fleetint-otelcol"
echo "ghcr_image=${GHCR_IMAGE}" >> "$GITHUB_OUTPUT"
echo "tags<<EOF" >> "$GITHUB_OUTPUT"
echo "${GHCR_IMAGE}:${VERSION}" >> "$GITHUB_OUTPUT"
if [ "${{ steps.version.outputs.is_prerelease }}" = "false" ]; then
echo "${GHCR_IMAGE}:latest" >> "$GITHUB_OUTPUT"
fi
echo "EOF" >> "$GITHUB_OUTPUT"

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

Expand Down Expand Up @@ -124,6 +139,15 @@ jobs:
REVISION=${{ steps.version.outputs.commit }}
BUILD_TIMESTAMP=${{ steps.version.outputs.build_timestamp }}

- name: Build and push fleetint-otelcol image
uses: docker/build-push-action@v6
with:
context: .
file: ./otelcol/Dockerfile
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.otelcol_image.outputs.tags }}

- name: Package Helm chart
run: |
helm package deployments/helm/fleet-intelligence-agent \
Expand Down Expand Up @@ -191,6 +215,9 @@ jobs:
echo "**fleet-intelligence-agent image tags:**" >> $GITHUB_STEP_SUMMARY
echo "${{ steps.agent_image_ghcr.outputs.tags }}" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**fleetint-otelcol image tags:**" >> $GITHUB_STEP_SUMMARY
echo "${{ steps.otelcol_image.outputs.tags }}" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Helm chart (GHCR OCI):**" >> $GITHUB_STEP_SUMMARY
echo "repo: oci://ghcr.io/${{ steps.agent_image_ghcr.outputs.owner_lc }}/charts" >> $GITHUB_STEP_SUMMARY
echo "chart: fleet-intelligence-agent:${{ steps.version.outputs.version }}" >> $GITHUB_STEP_SUMMARY
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,4 @@ AGENTS.md

.worktrees/
docs/plans/
otelcol/bin/
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ ROOTDIR=$(dir $(abspath $(lastword $(MAKEFILE_LIST))))

BUILD_TIMESTAMP ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
VERSION ?= $(shell git describe --match 'v[0-9]*' --dirty='.m' --always)
REVISION=$(shell git rev-parse HEAD)$(shell if ! git diff --no-ext-diff --quiet --exit-code; then echo .m; fi)
REVISION ?= $(shell git rev-parse HEAD)$(shell if ! git diff --no-ext-diff --quiet --exit-code; then echo .m; fi)
PACKAGE=github.com/NVIDIA/fleet-intelligence-agent

ifneq "$(strip $(shell command -v $(GO) 2>/dev/null))" ""
Expand Down Expand Up @@ -118,7 +118,7 @@ docker-test: ## build test image and run tests in container
-t $(TEST_IMAGE) \
.
@echo "Running tests..."
@$(DOCKER) run --rm $(TEST_IMAGE)
@$(DOCKER) run --rm -e VERSION=$(VERSION) -e REVISION=$(REVISION) $(TEST_IMAGE)

# Specific target for fleetint (your main binary)
fleetint: bin/fleetint ## build fleetint binary
Expand Down
51 changes: 51 additions & 0 deletions cmd/fleetint/gateway_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"testing"

pkgmetadata "github.com/NVIDIA/fleet-intelligence-sdk/pkg/metadata"
"github.com/stretchr/testify/require"

"github.com/NVIDIA/fleet-intelligence-agent/internal/config"
)

func TestConfigureHealthExporterFromEnvCollectorEndpoint(t *testing.T) {
t.Setenv("FLEETINT_COLLECTOR_ENDPOINT", "http://fleetint-otel-gateway:4318")
cfg := &config.Config{HealthExporter: &config.HealthExporterConfig{}}

require.NoError(t, configureHealthExporterFromEnv(cfg))
require.Equal(t, "http://fleetint-otel-gateway:4318", cfg.HealthExporter.CollectorEndpoint)
}

func TestConfigureHealthExporterFromEnvPreservesCollectorEndpointWhenUnset(t *testing.T) {
t.Setenv("FLEETINT_COLLECTOR_ENDPOINT", "")
cfg := &config.Config{HealthExporter: &config.HealthExporterConfig{
CollectorEndpoint: "https://collector.example",
}}

require.NoError(t, configureHealthExporterFromEnv(cfg))
require.Equal(t, "https://collector.example", cfg.HealthExporter.CollectorEndpoint)
}

func TestMaskMetadataValue(t *testing.T) {
const secret = "secret-token-value"

require.Equal(t, pkgmetadata.MaskToken(secret), maskMetadataValue(pkgmetadata.MetadataKeyToken, secret))
require.Equal(t, pkgmetadata.MaskToken(secret), maskMetadataValue("sak_token", secret))
require.Equal(t, secret, maskMetadataValue("backend_base_url", secret))
}
13 changes: 8 additions & 5 deletions cmd/fleetint/metadata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,7 @@ func metadataCommand(cliContext *cli.Context) error {
log.Logger.Debugw("successfully read metadata")

for k, v := range metadata {
// Mask sensitive tokens (JWT and SAK)
if k == pkgmetadata.MetadataKeyToken || k == "sak_token" {
v = pkgmetadata.MaskToken(v)
}
fmt.Printf("%s: %s\n", k, v)
fmt.Printf("%s: %s\n", k, maskMetadataValue(k, v))
}

setKey := cliContext.String("set-key")
Expand Down Expand Up @@ -114,3 +110,10 @@ func metadataCommand(cliContext *cli.Context) error {
fmt.Printf("%s successfully updated metadata\n", cmdutil.CheckMark)
return nil
}

func maskMetadataValue(key, value string) string {
if key == pkgmetadata.MetadataKeyToken || key == "sak_token" {
return pkgmetadata.MaskToken(value)
}
return value
}
8 changes: 8 additions & 0 deletions cmd/fleetint/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,14 @@ func configureHealthExporterFromEnv(cfg *config.Config) error {
return err
}

// OTel gateway collector mode: when set, metrics/logs are routed to the gateway
// instead of the backend directly. Enrollment remains independent for inventory
// and attestation; backend credentials are not forwarded to the gateway.
if val := os.Getenv("FLEETINT_COLLECTOR_ENDPOINT"); val != "" {
he.CollectorEndpoint = val
log.Logger.Infow("set OTel gateway collector endpoint from env", "collector_endpoint", val)
Comment on lines +225 to +227

@coderabbitai coderabbitai Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Do not log the raw collector endpoint.

FLEETINT_COLLECTOR_ENDPOINT is runtime-controlled, but the full value is written to logs. A misconfigured value containing userinfo, a token in the path, or other sensitive material would leak into application logs. Log only that the override was enabled, or emit a safely redacted value.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/fleetint/run.go` around lines 225 - 227, Update the
FLEETINT_COLLECTOR_ENDPOINT handling in the environment override block so Infow
does not log the raw endpoint value. Log only that the collector endpoint
override was enabled, or pass a safely redacted endpoint while preserving
assignment of the full value to he.CollectorEndpoint.

}

return nil
}

Expand Down
4 changes: 4 additions & 0 deletions deployments/helm/fleet-intelligence-agent/templates/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,10 @@ spec:
fieldPath: spec.nodeName
- name: IS_RUNTIME_K8S
value: "true"
{{- if .Values.otelGateway.enabled }}
- name: FLEETINT_COLLECTOR_ENDPOINT
value: {{ printf "http://%s-otel-gateway:%d" (include "fleet-intelligence-agent.fullname" .) (int .Values.otelGateway.service.port) | quote }}
{{- end }}
ports:
- name: http
containerPort: {{ .Values.ports.http }}
Expand Down
85 changes: 85 additions & 0 deletions deployments/helm/fleet-intelligence-agent/templates/otel-gateway-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

{{- if .Values.otelGateway.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "fleet-intelligence-agent.fullname" . }}-otel-gateway
labels:
{{- include "fleet-intelligence-agent.labels" . | nindent 4 }}
data:
config.yaml: |
extensions:
# Fleet Intelligence auth: enrolls with the backend using the SAK token, extracts
# the customer ID from the returned JWT, and refreshes the JWT proactively via
# response headers and on 401 as a fallback.
sakauth:
enroll_endpoint: ${env:ENROLL_ENDPOINT}
sak_token: ${env:SAK_TOKEN}
health_check:
endpoint: 0.0.0.0:13133

receivers:
otlp:
protocols:
http:
# No inbound auth: the gateway is a ClusterIP service accessible only
# within the cluster. Cluster network isolation is the security boundary.
# The gateway uses its own SAK JWT when forwarding to the backend.
# Agent identity flows through OTLP resource attributes (machine.id, GPU UUIDs).
endpoint: 0.0.0.0:{{ .Values.otelGateway.service.port }}

processors:
memory_limiter:

@ambermingxin ambermingxin Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from default values.yaml,

memory_limiter:
  limit_percentage: 80
  spike_limit_percentage: 20

is this limiter computed from the total node mem or the pod limit? if it's the former, there could possibly be OOM issue if the pod is capped at a smaller mem e.g. resources.limits.memory: 512Mi while the node itself is large, say 512G

check_interval: 5s
limit_percentage: {{ .Values.otelGateway.memoryLimiter.limitPercentage }}
spike_limit_percentage: {{ .Values.otelGateway.memoryLimiter.spikeLimitPercentage }}
batch:
timeout: 10s
send_batch_size: 1000

exporters:
otlp_http/backend:
metrics_endpoint: ${env:BACKEND_ENDPOINT}/metrics
logs_endpoint: ${env:BACKEND_ENDPOINT}/logs
compression: none
auth:
authenticator: sakauth
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 4
queue_size: 1000

service:
extensions: [sakauth, health_check]
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp_http/backend]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp_http/backend]
telemetry:
logs:
level: warn
{{- end }}
Loading
Loading