Skip to content

fix(worker-llm-credentials): write worker-token 0644 so pylon can read it#426

Merged
Max-NV merged 1 commit into
mainfrom
fix/worker-token-perms
Jul 24, 2026
Merged

fix(worker-llm-credentials): write worker-token 0644 so pylon can read it#426
Max-NV merged 1 commit into
mainfrom
fix/worker-token-perms

Conversation

@Max-NV

@Max-NV Max-NV commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Why

Managed LLM workers never register with the request router, so gateway invokes return no_eligible_candidates. The credential-manager writes /var/run/llm/worker-token as 0600 (owner root), but pylon (llm-router-client) reads it via --auth-token-file while running as a different non-root UID (nvs) on the shared emptyDir, so it gets EACCES and never authenticates its registration.

What changed

Write the worker-token 0644 so the pylon sidecar can read it. The llm emptyDir is mounted only by the two infra sidecars (llm-router-client + llm-credential-manager), not the customer workload container, so this does not expose the token to customer code.

Security note: this makes a router-auth token world-readable within the pod (gosec G306, suppressed with a reason). The alternative is a matched runAsUser + 0600 rendered by icms-translate (#409); this PR is the simpler, credential-manager-local fix. Pick one.

Customer Release Notes

LLM function workers now register with the request router in managed deployments; OpenAI-compatible invocations no longer fail with no_eligible_candidates due to the worker-token permission mismatch.

Testing

go test ./internal/worker/ (TestRun_CreatesMissingTokenDir now asserts the token file is written 0644).

References

Closes #410

Summary by CodeRabbit

  • Bug Fixes

    • Updated refreshed worker token persistence to use 0644 permissions so required supporting processes can read the shared token file.
    • Ensured the token contents remain unchanged while applying the expected permission bits.
  • Tests

    • Strengthened the token-file test to set a restrictive umask and verify the token directory is created and the file permissions are exactly 0644.

@Max-NV
Max-NV requested a review from a team as a code owner July 24, 2026 18:28
@Max-NV
Max-NV requested a review from shobham-nv July 24, 2026 18:28
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The worker now persists refreshed tokens with 0644 permissions, explicitly applies the mode before renaming, and documents the sidecar access requirement. The coverage test validates exact permissions under a restrictive umask.

Changes

Worker token permissions

Layer / File(s) Summary
Token write and permission validation
src/compute-plane-services/worker-llm-credentials/internal/worker/worker.go, src/compute-plane-services/worker-llm-credentials/internal/worker/worker_coverage_test.go
Token writes use mode 0644 and explicit chmod, with GoSec annotations; the coverage test applies umask 077 and asserts the persisted file mode.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • NVIDIA/nvcf#409: Adjusts LLM sidecar security contexts to share access to the worker token.

Suggested reviewers: shobham-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The change addresses issue #410 by making the worker token readable by the pylon sidecar and adding coverage for the new mode.
Out of Scope Changes check ✅ Passed The diff stays focused on worker-token permissions and the corresponding test update, with no unrelated changes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title follows Conventional Commits and accurately describes the bug fix to worker-token file permissions.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/worker-token-perms

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/compute-plane-services/worker-llm-credentials/internal/worker/worker.go`:
- Around line 101-105: Ensure the token refresh flow explicitly applies mode
0644 to the temporary file with os.Chmod after os.WriteFile and before
os.Rename, including when the temporary file already exists. Handle Chmod errors
consistently with the existing write/rename errors, and add tests covering
restrictive umasks and pre-existing temporary files.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 89a0c411-9f39-4f4a-a441-cd6be37ef6af

📥 Commits

Reviewing files that changed from the base of the PR and between cdde9cc and 4e12b4c.

📒 Files selected for processing (2)
  • src/compute-plane-services/worker-llm-credentials/internal/worker/worker.go
  • src/compute-plane-services/worker-llm-credentials/internal/worker/worker_coverage_test.go

…d it

Pylon (llm-router-client) reads /var/run/llm/worker-token to authenticate
its router registration, but it runs as a different non-root UID than the
credential-manager (root) on the shared emptyDir, so it could not read the
0600 token. Registration failed and gateway invokes returned
no_eligible_candidates.

Write the token 0644 and chmod the tmp file to force the mode: os.WriteFile
honors umask and skips the mode when tmp already exists, so a restrictive
container umask or a stale tmp could otherwise leave it 0600. The llm
emptyDir is mounted only by the two infra sidecars, not the customer
workload container.

NO-REF
@Max-NV
Max-NV force-pushed the fix/worker-token-perms branch from 4e12b4c to 04616dc Compare July 24, 2026 18:38

@estroz estroz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Max-NV
Max-NV enabled auto-merge July 24, 2026 18:48
@Max-NV Max-NV self-assigned this Jul 24, 2026
@Max-NV
Max-NV removed the request for review from shobham-nv July 24, 2026 18:55
@Max-NV
Max-NV disabled auto-merge July 24, 2026 19:32
@Max-NV
Max-NV added this pull request to the merge queue Jul 24, 2026
Merged via the queue into main with commit 698e3e9 Jul 24, 2026
17 checks passed
@Max-NV
Max-NV deleted the fix/worker-token-perms branch July 24, 2026 19:56
@balajinvda

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version nvcf-worker-llm-credentials-v1.0.6 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM workers fail to register: pylon cannot read the 0600 worker-token (sidecar UID mismatch)

3 participants