Skip to content

nix: extract K8s manifests, remove BCC, improve build isolation#2516

Open
randomizedcoder wants to merge 1 commit intoperformancecopilot:mainfrom
randomizedcoder:nix_extract_manifests
Open

nix: extract K8s manifests, remove BCC, improve build isolation#2516
randomizedcoder wants to merge 1 commit intoperformancecopilot:mainfrom
randomizedcoder:nix_extract_manifests

Conversation

@randomizedcoder
Copy link
Contributor

Standalone K8s manifest generation (nix build .#pcp-k8s-manifests) so users can inspect or deploy PCP's DaemonSet without minikube. Manifests are validated at build time with kube-linter; all 9 default findings are suppressed with documented rationale (PCP is a privileged node monitoring agent that needs host PID, /proc, /sys, and BPF access by design).

The nix/k8s-test module now imports from nix/k8s-manifests with a namespace override ("pcp-test") for test isolation, replacing the old nix/k8s-test/manifests.nix.

Completely remove BCC (pmdabcc was removed from PCP upstream in commit 9a576b8). This drops the unused bcc build dependency from nix/package.nix, invalid --with-pmdabcc configure flags, the dead nix/bcc.nix NixOS module (304 lines), and all enableBcc plumbing from nix/microvm.nix and flake.nix.

Pin minikube's --kubernetes-version to v1.33.0 so the test suite doesn't break when nixpkgs updates minikube. The start script now auto-deletes stale clusters with incompatible k8s versions instead of failing.

Exclude nix/, flake.nix, flake.lock, and CLAUDE.md from the PCP source filter so that Nix infrastructure changes don't trigger a full PCP rebuild.

Standalone K8s manifest generation (nix build .#pcp-k8s-manifests)
so users can inspect or deploy PCP's DaemonSet without minikube.
Manifests are validated at build time with kube-linter; all 9
default findings are suppressed with documented rationale (PCP is
a privileged node monitoring agent that needs host PID, /proc,
/sys, and BPF access by design).

The nix/k8s-test module now imports from nix/k8s-manifests with a
namespace override ("pcp-test") for test isolation, replacing the
old nix/k8s-test/manifests.nix.

Completely remove BCC (pmdabcc was removed from PCP upstream in
commit 9a576b8).  This drops the unused bcc build dependency
from nix/package.nix, invalid --with-pmdabcc configure flags,
the dead nix/bcc.nix NixOS module (304 lines), and all enableBcc
plumbing from nix/microvm.nix and flake.nix.

Pin minikube's --kubernetes-version to v1.33.0 so the test suite
doesn't break when nixpkgs updates minikube.  The start script
now auto-deletes stale clusters with incompatible k8s versions
instead of failing.

Exclude nix/, flake.nix, flake.lock, and CLAUDE.md from the PCP
source filter so that Nix infrastructure changes don't trigger a
full PCP rebuild.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@randomizedcoder
Copy link
Contributor Author

Generates the manifests. Also runs the manifest lint

[das@l:~/Downloads/pcp]$ nix build .#pcp-k8s-manifests
warning: Git tree '/home/das/Downloads/pcp' is dirty

[das@l:~/Downloads/pcp]$ ls ./result/
combined.yaml  daemonset.yaml  namespace.yaml  README.md

[das@l:~/Downloads/pcp]$ cat ./result/*
apiVersion: v1
kind: Namespace
metadata:
  name: pcp
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: pcp
  namespace: pcp
  labels:
    app: pcp
spec:
  selector:
    matchLabels:
      app: pcp
  template:
    metadata:
      labels:
        app: pcp
    spec:
      # Required for seeing all node processes
      hostPID: true

      containers:
      - name: pcp
        image: pcp:latest
        imagePullPolicy: Never

        # Privileged for BPF and full /proc access
        # Run as root to override container's default pcp user
        securityContext:
          privileged: true
          runAsUser: 0

        ports:
        - containerPort: 44321
          name: pmcd
        - containerPort: 44322
          name: pmproxy

        env:
        # Tell PCP where host filesystem is mounted
        - name: HOST_MOUNT
          value: "/host"
        - name: PCP_SYSFS_DIR
          value: "/host/sys"

        volumeMounts:
        # Host root filesystem (read-only)
        - name: host-root
          mountPath: /host
          readOnly: true
        # Required for BPF
        - name: sys-kernel-debug
          mountPath: /sys/kernel/debug
        # Host /proc for process metrics
        - name: host-proc
          mountPath: /host/proc
          readOnly: true
        # Host /sys for system metrics
        - name: host-sys
          mountPath: /host/sys
          readOnly: true

        readinessProbe:
          tcpSocket:
            port: pmcd
          initialDelaySeconds: 10
          periodSeconds: 5

        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "100m"

      volumes:
      - name: host-root
        hostPath:
          path: /
      - name: sys-kernel-debug
        hostPath:
          path: /sys/kernel/debug
      - name: host-proc
        hostPath:
          path: /proc
      - name: host-sys
        hostPath:
          path: /sys

      # Run on all nodes including control plane
      tolerations:
      - operator: Exists
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: pcp
  namespace: pcp
  labels:
    app: pcp
spec:
  selector:
    matchLabels:
      app: pcp
  template:
    metadata:
      labels:
        app: pcp
    spec:
      # Required for seeing all node processes
      hostPID: true

      containers:
      - name: pcp
        image: pcp:latest
        imagePullPolicy: Never

        # Privileged for BPF and full /proc access
        # Run as root to override container's default pcp user
        securityContext:
          privileged: true
          runAsUser: 0

        ports:
        - containerPort: 44321
          name: pmcd
        - containerPort: 44322
          name: pmproxy

        env:
        # Tell PCP where host filesystem is mounted
        - name: HOST_MOUNT
          value: "/host"
        - name: PCP_SYSFS_DIR
          value: "/host/sys"

        volumeMounts:
        # Host root filesystem (read-only)
        - name: host-root
          mountPath: /host
          readOnly: true
        # Required for BPF
        - name: sys-kernel-debug
          mountPath: /sys/kernel/debug
        # Host /proc for process metrics
        - name: host-proc
          mountPath: /host/proc
          readOnly: true
        # Host /sys for system metrics
        - name: host-sys
          mountPath: /host/sys
          readOnly: true

        readinessProbe:
          tcpSocket:
            port: pmcd
          initialDelaySeconds: 10
          periodSeconds: 5

        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "100m"

      volumes:
      - name: host-root
        hostPath:
          path: /
      - name: sys-kernel-debug
        hostPath:
          path: /sys/kernel/debug
      - name: host-proc
        hostPath:
          path: /proc
      - name: host-sys
        hostPath:
          path: /sys

      # Run on all nodes including control plane
      tolerations:
      - operator: Exists
apiVersion: v1
kind: Namespace
metadata:
  name: pcp
PCP Kubernetes Manifests
========================

These manifests deploy PCP as a privileged DaemonSet for full node
monitoring including BPF metrics.

Namespace: pcp

Quick start:

  # Apply all manifests at once
  kubectl apply -f combined.yaml

  # Or apply individually
  kubectl apply -f namespace.yaml
  kubectl apply -f daemonset.yaml

  # Verify deployment
  kubectl get pods -n pcp

  # Query metrics from a pod
  kubectl exec -n pcp <pod> -- pminfo -f kernel.all.load

  # Cleanup
  kubectl delete namespace pcp

Generated by: nix build .#pcp-k8s-manifests

[das@l:~/Downloads/pcp]$ 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant