diff --git a/src/experimental/dx-standalone-linux/README.md b/src/experimental/dx-standalone-linux/README.md new file mode 100644 index 00000000000..19b245476ba --- /dev/null +++ b/src/experimental/dx-standalone-linux/README.md @@ -0,0 +1,193 @@ +# dx on pure Linux — standalone PEM-direct (no Kubernetes) + +*Showcase artifact (fork-only, excluded from copybara — not for upstream). Runs +the **dx active-diagnosis daemon** against a **standalone Pixie PEM** on a single +Linux VM, with **no Kubernetes**, and shows three ways to keep **Pixie Cloud +visualization** working when the PEM has no vizier middleware.* + +The companion effort estimate is `dx: docs/DEPLOYMENT_ALTERNATIVES.md` (Q2). This +directory is the buildable/testable realization of it. + +--- + +## 1. What runs, and what each piece is for + +``` +┌────────────────────────────── one Linux VM ─────────────────────────────┐ +│ │ +│ standalone_pem (src/experimental/standalone_pem, systemd, privileged) │ +│ eBPF capture (Stirling) + in-memory table store + ExecuteScript gRPC │ +│ on :12345. NO cloud-connector, NO query-broker, NO metadata service. │ +│ ▲ pxapi ExecuteScript (node-local gRPC) │ +│ │ │ +│ dx-daemon (entlein/dx, systemd) │ +│ DX_BENCH=pxdirect PX_DIRECT_ADDR=127.0.0.1:12345 │ +│ S2 receiver :9099 ← referrals (no kubescape on a VM — see §3) │ +│ triage → workup (queries the PEM) → belief verdict → /metrics :9095 │ +│ │ +│ [optional] cloud path for visualization — §4: │ +│ A) cloud-connector-shim (this dir, systemd) → Pixie Cloud vzconn │ +│ B) vizier + kelvin as systemd middleware → Pixie Cloud vzconn │ +└────────────────────────────────────────────────────────────────────────────┘ +``` + +**Detection needs no cloud.** dx queries the PEM directly over `pxdirect`; the +whole diagnostic loop is node-local. Pixie Cloud is only for the **human +visualization / PxL UI**, and §4 is how you keep it. + +--- + +## 2. Quick start (detection path, no cloud) + +### 2a. docker-compose (fastest) +``` +cd compose && sudo docker compose up -d +# standalone_pem (privileged, hostPID, /sys + host-root) + dx-daemon on host net +``` + +### 2b. systemd (the "pure Linux" target) +``` +sudo cp systemd/*.service /etc/systemd/system/ +sudo cp systemd/dx-standalone.env /etc/dx/ # edit paths/addrs +sudo systemctl daemon-reload +sudo systemctl enable --now standalone-pem dx-daemon +journalctl -u dx-daemon -f | grep 'catalog loaded' # → 12 conditions +``` +Prereqs: a kernel with BTF/CO-RE (or matching headers) for the PEM's eBPF probes; +root/`CAP_SYS_ADMIN`+`CAP_BPF`+`CAP_SYS_PTRACE`; host PID namespace. The +`standalone-pem.service` declares exactly these. + +### 2c. verify +``` +./inject/inject-referral.sh argocd-render # a synthetic kubescape finding +curl -s localhost:9095/metrics | grep dx_verdicts_total +journalctl -u dx-daemon | grep ruled_in +``` + +--- + +## 3. Referrals without kubescape (the "no k8s" input) + +On k8s, dx's referrals are kubescape runtime alerts. On a bare VM there is no +kubescape node-agent, so referrals come from one of: + +1. **The injector (`inject/inject-referral.sh`)** — POSTs a synthetic kubescape- + shaped finding to dx's S2 receiver (`:9099/findings`). Used by the tests + the + showcase; also the shape any real feed must emit. +2. **A referral synthesizer from the PEM stream** (the recommended production + path, DEPLOYMENT_ALTERNATIVES §4.4): derive R00xx-equivalent referrals from the + PEM's own eBPF `process_stats`/`conn_stats`/file events + operator-authored + baselines — one eBPF agent, no kubescape. Sketched in §6; not built here. + +dx carries enough **event-driven evidence in the referral itself** (spawn marker, +file path, the kernel-evidence markers) that a rich referral produces a verdict +**even before the PEM pull** — which is why the smoke test works without eBPF. + +--- + +## 4. Keeping Pixie Cloud visualization — the cloud-auth problem + +**The gap.** Pixie Cloud reaches a cluster through the vizier **cloud-connector**, +which opens an mTLS tunnel to the cloud **vzconn** service +(`src/cloud/vzconn`), registered with a **deploy key** → cluster gets an ID + a +cloud-issued cert; the UI's PxL queries are then tunneled cloud → cloud-connector +→ query-broker → PEMs. `standalone_pem` has **none** of cloud-connector, +query-broker, or vzconn — so **Pixie Cloud cannot see it**. Three ways to fix it, +cheapest first: + +### Option A — cloud-connector-shim (recommended for a single VM) +A small Go sidecar (`cloudshim/`, systemd `cloud-connector-shim.service`) that: +1. **Authenticates to cloud** exactly as the real cloud-connector does — dial + `vzconn` with the **deploy key** over mTLS (`vzconn_client.go` flow: + `RegisterVizier` → receive cluster ID + SSL certs → maintain the tunnel). +2. **Proxies queries** — a cloud-tunneled `ExecuteScript` is forwarded to the + local `standalone_pem` on `:12345` and the result batches streamed back up the + tunnel. + +This keeps the PEM binary unchanged and collapses cloud-connector + query-broker +into one shim (single node → no Kelvin fan-out needed). **Cloud shows the cluster +and runs PxL; the dx PoC keeps its own node-local pxdirect path in parallel.** +- *Auth detail:* the deploy key + `PL_CLUSTER_ID` come from cloud + (`px deploy-key create`); the mTLS handshake is the same brittle step as a + self-hosted-cloud stand-up (DEPLOYMENT_ALTERNATIVES Q1) — get the DNS/cert chain + right and it connects. +- *Metadata caveat (below §4-meta) applies:* PxL `upid_to_pod_name` etc. resolve + to empty; the UI shows process/cgroup identity, not pods. +- Effort: the registration handshake + the ExecuteScript proxy; skeleton + + precise wiring points are in `cloudshim/` (this dir). + +### Option B — run vizier + Kelvin as systemd middleware +Run the real control plane off-k8s as systemd units (`systemd/middleware/`): +`pl-nats`, `pl-etcd`, `vizier-metadata`, `vizier-query-broker`, `vizier-kelvin`, +`vizier-cloud-connector`, with a **real** PEM as the agent (registers via +NATS/metadata). Cloud sees a fully normal vizier; Kelvin gives cross-query exec. +- This is the **fullest** cloud experience and the **heaviest**: vizier's metadata + service is built to watch the **k8s API** for the upid→pod map + (`metadata/controllers/k8smeta`). Off k8s you run it in a **degraded metadata** + mode (static host/cgroup map) — topology columns are synthetic. NATS+etcd+4 + services is a real resident footprint on one VM. +- Unit templates + the metadata caveat are in `systemd/middleware/README.md`. + +### Option C — no cloud (detection only) +Skip cloud entirely; visualize from dx's own outputs — the `/metrics` scrape +(Prometheus/Grafana), the structured verdict log, and the evidence manifests. +This is §2 and needs nothing extra. Recommended when the VM has no route to a +Pixie Cloud. + +### §4-meta — the metadata degradation (applies to A and B) +`upid_to_pod_name` / `upid_to_namespace` / `upid_to_service_name` are computed by +the metadata service watching the **k8s API**; on a VM there are no pods, so they +return empty. dx already tolerates this (it can key on process/cgroup identity — +DEPLOYMENT_ALTERNATIVES §3.2.3). For the **cloud UI**, PxL scripts that group by +`pod`/`service` show blank; a VM-native PxL uses `upid`, `cmdline`, cgroup path. +A static host→service map (a config file the shim/metadata reads) restores +readable names at lower fidelity. + +**Recommendation:** Option A for a single showcase VM (cloud viz + minimal +footprint), Option C when there is no cloud, Option B only if full k8s-style +topology in the UI is a hard requirement. + +--- + +## 5. Tests & NFRs (what runs here vs. on a real kernel) + +- `test/smoke.sh` — brings up dx (PEM optional), injects a known attack referral, + asserts a malignant verdict + the exported metrics. Runs on **any** Linux + (no eBPF needed — event-driven referral evidence, §3). +- `test/nfr.sh` — the single-VM NFR harness: drives N referrals through dx and + reports p50/p95 **time-to-verdict** (`dx_time_to_verdict_seconds`), throughput, + backpressure drops, and **RSS/CPU** (`process_resident_memory_bytes`, + `process_cpu_seconds_total`) — the pure-Linux equivalent of the k8s report G1/G2. +- The **PEM eBPF path** (real `conn_stats`/`http_events` pulls via pxdirect) needs + a privileged PEM on a CO-RE kernel; the harness exercises it when a PEM is + present and degrades to referral-only otherwise (never a silent pass). + +### 5a. Validated on a CO-RE VM (2026-07-04, kernel 6.8, BTF present) +Ran the **real** `standalone_pem` image + dx on this host, no k8s/cloud: +- PEM: Stirling attached `socket_tracer` and registered `conn_stats`, + `http_events`, `dns_events` (+ mysql/redis/mongo/amqp/tls) — **only with + `/sys` mounted read-write** (read-only blocks kprobe attach: + `probe cleanup failed: /sys/kernel/debug/tracing/kprobe_events`). The compose + + `standalone-pem.service` mount `/sys:rw` for this reason. +- dx `pxdirect` executed PxL against the PEM: `dx_bench_errors_total 0`, + `dx_bench_unavailable 0`, tables resolve (no "Table not found"), a triage pull + returned in ~12 ms (`"dur_ms":12,"err":false`). Referrals admitted, verdicts + produced (`generic=MALIGNANT`). +- **The off-k8s caveat, observed directly:** a referral scoped to a synthetic pod + (`namespace=poc`) pulls **0 rows** because `upid_to_namespace` is empty off-k8s + and the `namespace=='poc'` filter matches nothing — exactly §4-meta. On a VM, + omit the pod scope (empty `RuntimeK8sDetails`) so dx queries the PEM unscoped, or + supply a static metadata map. This is a real behaviour, not a bug. + +See `test/README.md` for exact invocations + pass criteria. + +--- + +## 6. Not built here (follow-ups) +- The PEM→referral **synthesizer** (§3.2) — one eBPF agent replacing kubescape. +- A **static metadata** provider (host/cgroup→service) for readable cloud names. +- Full `cloudshim` ExecuteScript proxy hardening + the vzconn cert rotation. + +Everything in this directory is **fork-only** and excluded from copybara +(`tools/private/copybara/copy.bara.sky`) — it packages an experimental, +non-upstream deployment mode of the PoC. diff --git a/src/experimental/dx-standalone-linux/cloudshim/README.md b/src/experimental/dx-standalone-linux/cloudshim/README.md new file mode 100644 index 00000000000..174fdad8779 --- /dev/null +++ b/src/experimental/dx-standalone-linux/cloudshim/README.md @@ -0,0 +1,38 @@ +# cloud-connector-shim — how the standalone PEM authenticates to Pixie Cloud + +The problem (README §4): Pixie Cloud reaches a cluster via the vizier +**cloud-connector** → cloud **vzconn** mTLS tunnel. `standalone_pem` has no +cloud-connector, so cloud cannot see it. This shim is the minimal cloud-connector +for a single PEM. + +## Auth flow (what `registerWithCloud` implements) +1. Operator: `px deploy-key create` → a deploy key; mount at `PL_DEPLOY_KEY_FILE`. +2. Shim dials `PL_CLOUD_ADDR` vzconn over **mTLS** and calls **RegisterVizier** + (`src/cloud/vzconn/vzconnpb`), passing the deploy key as the JWT + a + `ClusterInfo{ClusterName,…}`. +3. Cloud assigns a **cluster ID** + issues **SSL certs**; the shim persists the ID + to `PL_CLUSTER_ID_FILE` and keeps the NATS-bridge tunnel open (heartbeats). + Mirror: `src/vizier/services/cloud_connector/bridge/vzconn_client.go`. +4. Pixie Cloud now lists the cluster; PxL from the UI is tunneled down. + +## Query proxy (what `serveProxy` implements) +A cloud-tunneled `ExecuteScript` is run against the local `standalone_pem` +(`PEM_ADDR`, :12345) via `px.dev/pixie/src/api/go/pxapi` (same client dx's +`pxdirect` uses) and the result RowBatches are streamed back up the tunnel. One +agent → no distributed plan / Kelvin needed. + +## Build / run +``` +go build -o dx-cloud-connector-shim . +PL_DEPLOY_KEY_FILE=/etc/dx/deploy.key PL_CLOUD_ADDR=withpixie.ai:443 \ + PEM_ADDR=127.0.0.1:12345 ./dx-cloud-connector-shim +``` + +## Status +This is a compiling **skeleton**: config + control loop are wired; the two +integration points (vzconn RegisterVizier, ExecuteScript proxy) are marked +against the real pixie packages and are the follow-up — they can only be +validated against a live Pixie Cloud + the vzconn cert chain (rig-gated). The +metadata caveat (README §4-meta) applies: PxL `upid_to_pod_name` returns empty +off-k8s; the UI shows process/cgroup identity unless a static host→service map is +supplied. diff --git a/src/experimental/dx-standalone-linux/cloudshim/main.go b/src/experimental/dx-standalone-linux/cloudshim/main.go new file mode 100644 index 00000000000..7195b82169d --- /dev/null +++ b/src/experimental/dx-standalone-linux/cloudshim/main.go @@ -0,0 +1,174 @@ +/* + * Copyright 2018- The Pixie Authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + * SPDX-License-Identifier: Apache-2.0 + */ + +// Command dx-cloud-connector-shim keeps Pixie Cloud VISUALIZATION working for a +// standalone PEM that has no vizier middleware (README §4, Option A). +// +// It does the two things the real vizier cloud-connector does, minus everything +// a single-node PoC does not need (no NATS, no metadata service, no Kelvin +// fan-out): +// +// 1. AUTHENTICATE the "cluster" (this one PEM) to Pixie Cloud — dial the cloud +// vzconn service with a DEPLOY KEY over mTLS, RegisterVizier, receive the +// cluster ID + cloud-issued SSL certs, and maintain the tunnel. This mirrors +// src/vizier/services/cloud_connector/bridge/vzconn_client.go + +// src/cloud/vzconn (RegisterVizierRequest → RegisterVizierResponse). +// +// 2. PROXY queries — a cloud-tunneled ExecuteScript is forwarded to the local +// standalone_pem on PEM_ADDR (:12345) via pxapi, and the result RowBatches +// are streamed back up the tunnel. This mirrors what query-broker does, but +// for a single agent (no distributed plan). +// +// This file is a COMPILING SKELETON (stdlib only): it wires config + the control +// loop and marks the two integration points against the real pixie packages. The +// full handshake/proxy is the follow-up (cloudshim/README.md) — it needs a live +// Pixie Cloud + the vzconn mTLS certs to validate, which is rig-gated. +// +// Fork-only, excluded from copybara. +package main + +import ( + "context" + "log" + "os" + "os/signal" + "syscall" + "time" +) + +type config struct { + cloudAddr string // PL_CLOUD_ADDR, e.g. withpixie.ai:443 + deployKey string // contents of PL_DEPLOY_KEY_FILE + clusterID string // PL_CLUSTER_ID_FILE (empty on first run → assigned by RegisterVizier) + pemAddr string // PEM_ADDR, the local standalone_pem ExecuteScript gRPC (:12345) + clusterName string // PL_CLUSTER_NAME (shown in the Pixie Cloud UI) +} + +func loadConfig() config { + read := func(path string) string { + if path == "" { + return "" + } + b, err := os.ReadFile(path) + if err != nil { + return "" + } + return string(b) + } + return config{ + cloudAddr: env("PL_CLOUD_ADDR", "withpixie.ai:443"), + deployKey: read(os.Getenv("PL_DEPLOY_KEY_FILE")), + clusterID: read(os.Getenv("PL_CLUSTER_ID_FILE")), + pemAddr: env("PEM_ADDR", "127.0.0.1:12345"), + clusterName: env("PL_CLUSTER_NAME", "dx-standalone-vm"), + } +} + +func env(k, def string) string { + if v := os.Getenv(k); v != "" { + return v + } + return def +} + +func main() { + log.SetFlags(log.LstdFlags | log.Lmsgprefix) + log.SetPrefix("[dx-cloud-shim] ") + cfg := loadConfig() + + if cfg.deployKey == "" { + log.Fatal("no deploy key (PL_DEPLOY_KEY_FILE): create one with `px deploy-key create` and mount it") + } + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT) + defer stop() + + log.Printf("cloud=%s pem=%s cluster_name=%q cluster_id=%q", + cfg.cloudAddr, cfg.pemAddr, cfg.clusterName, redact(cfg.clusterID)) + + // ── (1) AUTHENTICATE to cloud ──────────────────────────────────────────── + // Implementation point: dial vzconn over mTLS and register. + // + // import cloudvzconnpb "px.dev/pixie/src/cloud/vzconn/vzconnpb" + // conn := grpc.Dial(cfg.cloudAddr, grpc.WithTransportCredentials(mtls(cfg.deployKey))) + // stream, _ := cloudvzconnpb.NewVZConnServiceClient(conn).NATSBridge(ctx) // the tunnel + // // RegisterVizierRequest{VizierID?, JWTKey(deployKey), ClusterInfo{ClusterName, ...}} + // // → RegisterVizierAck{Status, VizierID, cloud-issued certs}. Persist VizierID + // // to PL_CLUSTER_ID_FILE. See cloud_connector/bridge/vzconn_client.go. + sess, err := registerWithCloud(ctx, cfg) + if err != nil { + log.Fatalf("cloud registration failed: %v", err) + } + log.Printf("registered with cloud: cluster_id=%s (Pixie Cloud can now see this PEM)", redact(sess.clusterID)) + + // ── (2) PROXY cloud-tunneled ExecuteScript → local PEM ─────────────────── + // Implementation point: for each cloud-tunneled query, run it against the + // local standalone_pem via pxapi and stream RowBatches back up the tunnel. + // + // import "px.dev/pixie/src/api/go/pxapi" + // pem, _ := pxapi.NewClient(ctx, pxapi.WithDirectAddr(cfg.pemAddr), + // pxapi.WithDirectCredsInsecure()) + // vz, _ := pem.NewVizierClient(ctx, "localhost") + // for msg := range sess.incomingQueries() { // from the vzconn tunnel + // vz.ExecuteScript(ctx, msg.PxL, tunnelCollector(sess, msg.QueryID)) + // } + if err := serveProxy(ctx, cfg, sess); err != nil && ctx.Err() == nil { + log.Fatalf("proxy loop: %v", err) + } + log.Print("shutting down") +} + +// ── skeleton internals (replace with the pixie-API wiring above) ───────────── + +type session struct{ clusterID string } + +// registerWithCloud performs the vzconn deploy-key mTLS registration. SKELETON: +// returns a session; the real handshake is the wiring in the (1) block. +func registerWithCloud(ctx context.Context, cfg config) (*session, error) { + id := cfg.clusterID + if id == "" { + // RegisterVizier assigns one on first connect; persist it for restarts. + id = "pending-register" + log.Print("no cluster id yet — RegisterVizier will assign one (persist to PL_CLUSTER_ID_FILE)") + } + // TODO(cloudshim): real vzconn dial + RegisterVizier (see (1)); until then this + // documents the flow and lets the unit start so the deployment is exercised. + return &session{clusterID: id}, nil +} + +// serveProxy forwards cloud-tunneled ExecuteScript to the local PEM. SKELETON: +// idles until ctx is done; the real loop is the wiring in the (2) block. +func serveProxy(ctx context.Context, cfg config, _ *session) error { + t := time.NewTicker(30 * time.Second) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + log.Printf("proxy alive — awaiting cloud queries, PEM at %s", cfg.pemAddr) + } + } +} + +func redact(s string) string { + if len(s) <= 8 { + return s + } + return s[:8] + "…" +} diff --git a/src/experimental/dx-standalone-linux/compose/docker-compose.yaml b/src/experimental/dx-standalone-linux/compose/docker-compose.yaml new file mode 100644 index 00000000000..e0eb60127b6 --- /dev/null +++ b/src/experimental/dx-standalone-linux/compose/docker-compose.yaml @@ -0,0 +1,44 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +--- +# Fast single-VM bring-up of the DETECTION path (no cloud, no k8s). Host network +# so dx reaches the PEM on 127.0.0.1:12345 and the injector reaches dx :9099. +services: + standalone-pem: + image: ghcr.io/k8sstormcenter/vizier-standalone_pem_image:0.14.18-dxpoc3 + network_mode: host + pid: host + privileged: true + environment: + PX_STANDALONE_PEM_PORT: "12345" + PL_TABLE_STORE_DATA_LIMIT_MB: "1024" + PL_HOST_PATH: /host + volumes: + - /:/host:ro + - /sys:/sys:rw + dx-daemon: + image: docker.io/entlein/dx-daemon:0.3.0-integration4 + network_mode: host + depends_on: + - standalone-pem + environment: + DX_PORT: "9099" + DX_METRICS_PORT: "9095" + DX_BENCH: pxdirect + PX_DIRECT_ADDR: 127.0.0.1:12345 + DX_CATALOG_DIR: /etc/dx/catalog + volumes: + - ./catalog:/etc/dx/catalog:ro diff --git a/src/experimental/dx-standalone-linux/inject/inject-referral.sh b/src/experimental/dx-standalone-linux/inject/inject-referral.sh new file mode 100755 index 00000000000..1148245af1e --- /dev/null +++ b/src/experimental/dx-standalone-linux/inject/inject-referral.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# inject-referral.sh — the "no kubescape on a VM" referral source. +# +# POSTs a synthetic, enriched-kubescape-shaped alert to the dx S2 receiver +# (:9099/findings, JSON array). This is the exact shape a real feed must emit +# (dx internal/receiver → referral.FromKubescapeRow). Used by the showcase + the +# smoke test; also a template for the PEM→referral synthesizer (README §3.2). +# +# Usage: inject-referral.sh [dx_url] +# scenarios: argocd-render | log4shell-spawn | cred-escalation | benign +# dx_url default http://127.0.0.1:9099 +set -euo pipefail +scenario="${1:-argocd-render}" +DX="${2:-http://127.0.0.1:9099}" +now="$(date +%s)" +ns="${NS-poc}" # NS="" → no pod scope (VM-native; dx queries the PEM unscoped) +pod="app-vm-0" + +# one enriched-kubescape row; RuntimeProcessDetails carries the process tree, +# RuntimeK8sDetails the (VM: synthetic) pod scope. event_time in seconds. +row() { # rule comm message [file] + local rule="$1" comm="$2" msg="$3" file="${4:-}" + cat < $DX/findings" +} + +case "$scenario" in + # completed argocd-style RCE: unexpected spawn (R0001) + sensitive-file read + # (R0010) on the same pod → dx correlates → argocd-malignant-render. + argocd-render) + post "[$(row R0001 mal.sh 'Unexpected process launched: mal.sh')]" + post "[$(row R0010 mal.sh 'Unexpected sensitive file access: /etc/shadow' /etc/shadow)]" + ;; + # log4shell contained-spawn: R0001 spawn (the JVM child). The http/ldap evidence + # would come from the PEM pull; here the spawn alone drives the invasion signal. + log4shell-spawn) + post "[$(row R0001 sh 'Unexpected process launched: sh (spawned by java)')]" + ;; + # privilege escalation via the credential subsystem (R0004 capabilities). + cred-escalation) + post "[$(row R0004 app 'Unexpected capabilities: CAP_SYS_ADMIN raised')]" + ;; + # a benign baseline referral (expected: not malignant / discharged). + benign) + post "[$(row R0002 sh 'File access: /var/log/app.log' /var/log/app.log)]" + ;; + *) echo "unknown scenario: $scenario (argocd-render|log4shell-spawn|cred-escalation|benign)"; exit 2;; +esac diff --git a/src/experimental/dx-standalone-linux/systemd/cloud-connector-shim.service b/src/experimental/dx-standalone-linux/systemd/cloud-connector-shim.service new file mode 100644 index 00000000000..c043060ace9 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/cloud-connector-shim.service @@ -0,0 +1,21 @@ +[Unit] +Description=dx cloud-connector shim (registers the standalone PEM with Pixie Cloud vzconn + proxies ExecuteScript) +After=standalone-pem.service network-online.target +Wants=standalone-pem.service + +[Service] +# Auth to cloud exactly as vizier's cloud-connector does: deploy key -> vzconn mTLS +# -> RegisterVizier -> cluster cert. Then proxy cloud-tunneled ExecuteScript to the +# local PEM :12345. See cloudshim/README.md for the wiring + what remains to build. +Environment=PL_CLOUD_ADDR=withpixie.ai:443 +Environment=PL_DEPLOY_KEY_FILE=/etc/dx/deploy.key +Environment=PL_CLUSTER_ID_FILE=/etc/dx/cluster.id +Environment=PEM_ADDR=127.0.0.1:12345 +ExecStart=/usr/local/bin/dx-cloud-connector-shim +DynamicUser=yes +NoNewPrivileges=yes +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/dx-daemon.service b/src/experimental/dx-standalone-linux/systemd/dx-daemon.service new file mode 100644 index 00000000000..02f8fe6966b --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/dx-daemon.service @@ -0,0 +1,23 @@ +[Unit] +Description=dx active-diagnosis daemon (node-local, PEM-direct) +After=standalone-pem.service network-online.target +Wants=standalone-pem.service + +[Service] +EnvironmentFile=/etc/dx/dx-standalone.env +ExecStart=/usr/local/bin/dx-daemon +# dx is unprivileged: it only talks gRPC to the local PEM and serves HTTP. +DynamicUser=yes +AmbientCapabilities= +NoNewPrivileges=yes +ProtectSystem=strict +ReadWritePaths=/var/lib/dx +MemoryMax=1G +CPUQuota=200% # the verdict-latency lever (report G2): 2000m→~120ms/verdict +Restart=on-failure +RestartSec=3 +# graceful drain (dx#8): dx flushes queued referrals on SIGTERM. +TimeoutStopSec=35 + +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/dx-standalone.env b/src/experimental/dx-standalone-linux/systemd/dx-standalone.env new file mode 100644 index 00000000000..3564aadb1ad --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/dx-standalone.env @@ -0,0 +1,13 @@ +# dx-daemon config for the single-VM PEM-direct deployment. Edit + copy to /etc/dx/. +# NOTE: systemd EnvironmentFile does NOT strip inline comments — a trailing +# "VALUE # note" is kept verbatim in the value. Keep every comment on its OWN line. +DX_PORT=9099 +DX_METRICS_PORT=9095 +DX_BENCH=pxdirect +PX_DIRECT_ADDR=127.0.0.1:12345 +# mount the 12-condition extended catalog at /etc/dx/catalog to enable it: +DX_CATALOG_DIR=/etc/dx/catalog +DX_WORKERS=4 +DX_PX_TIMEOUT_S=90 +# operator-authored behaviour baseline (optional): +# DX_SBOB=/etc/dx/sbob.json diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/README.md b/src/experimental/dx-standalone-linux/systemd/middleware/README.md new file mode 100644 index 00000000000..3799a36ba54 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/README.md @@ -0,0 +1,23 @@ +# Option B — vizier + Kelvin as systemd (full Pixie Cloud visualization off k8s) + +Run the control plane as systemd units so Pixie Cloud sees a normal vizier and +Kelvin gives cross-query exec. Heaviest option; use only if k8s-style topology in +the UI is required. Units are TEMPLATES — the images/binaries + config come from +the pixie build. + +Services (start in order): pl-etcd → pl-nats → vizier-metadata → +vizier-query-broker → vizier-kelvin → vizier-cloud-connector, with a real PEM +(not standalone_pem) as the agent so it registers via NATS/metadata. + +THE CAVEAT: vizier's metadata service (src/vizier/services/metadata/controllers/ +k8smeta) is built to watch the k8s API for the upid->pod/service/namespace map. +Off k8s there are no pods, so you run metadata in a DEGRADED mode backed by a +STATIC host/cgroup->service file (metadata-static.json). PxL topology columns +(pod, service, node) are then synthetic; group-by-pod in the UI shows the static +names, not real k8s objects. This is the price of dropping k8s (see the dx repo +docs/DEPLOYMENT_ALTERNATIVES.md §3.4). + +Each unit: Environment=PL_CLOUD_ADDR/PL_DEPLOY_KEY, After= the previous, Restart= +on-failure, MemoryMax set (etcd/nats/metadata are the stateful footprint). Wire +the cloud-connector's vzconn deploy-key registration the same way as the shim +(cloudshim/README.md). diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/middleware.env b/src/experimental/dx-standalone-linux/systemd/middleware/middleware.env new file mode 100644 index 00000000000..a3c1fe9719f --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/middleware.env @@ -0,0 +1,29 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# Shared config for the vizier+Kelvin middleware run as systemd (README Option B). +# These replace the values the k8s Deployments inject; edit + copy to /etc/vizier/. +PL_CLOUD_ADDR=withpixie.ai:443 +PL_JWT_SIGNING_KEY=CHANGE_ME_shared_hs256_key # same key across ALL vizier units +PL_DEPLOY_KEY=CHANGE_ME_from_px_deploy-key_create +PL_CLUSTER_ID= # assigned by cloud-connector on first RegisterVizier +PL_VIZIER_NAME=dx-standalone-vm +PL_POD_NAMESPACE=pl # a label only (no k8s); keeps PxL/UI names stable +PL_MD_ETCD_SERVER=http://127.0.0.1:2379 +PL_ETCD_OPERATOR_ENABLED=false +PL_HOST_PATH=/ +PL_HOST_IP=127.0.0.1 +PL_POD_IP_ADDRESS=127.0.0.1 +PL_DATA_ACCESS=Full diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/pl-etcd.service b/src/experimental/dx-standalone-linux/systemd/middleware/pl-etcd.service new file mode 100644 index 00000000000..80fcadf1ba5 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/pl-etcd.service @@ -0,0 +1,24 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=pl-etcd — metadata store for vizier-metadata (replaces the k8s etcd operator) +After=network-online.target +[Service] +ExecStart=/usr/local/bin/etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/vizier/etcd +Restart=on-failure +RestartSec=2 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/pl-nats.service b/src/experimental/dx-standalone-linux/systemd/middleware/pl-nats.service new file mode 100644 index 00000000000..f2f77899eb1 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/pl-nats.service @@ -0,0 +1,25 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=pl-nats — NATS message bus for the vizier middleware +After=network-online.target +Wants=network-online.target +[Service] +ExecStart=/usr/local/bin/nats-server -p 4222 -m 8222 +Restart=on-failure +RestartSec=2 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/vizier-cloud-connector.service b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-cloud-connector.service new file mode 100644 index 00000000000..aa073937a92 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-cloud-connector.service @@ -0,0 +1,29 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=vizier-cloud-connector — the Pixie Cloud tunnel (deploy-key -> vzconn mTLS RegisterVizier) +After=pl-nats.service vizier-query-broker.service +Requires=vizier-query-broker.service +[Service] +EnvironmentFile=/etc/vizier/middleware.env +# PL_DEPLOY_KEY (from `px deploy-key create`) authenticates to PL_CLOUD_ADDR vzconn; +# RegisterVizier returns the cluster id + certs; the tunnel then carries the UI's +# PxL down to query-broker. Port :50800. Persist the assigned PL_CLUSTER_ID. +ExecStart=/usr/local/bin/vizier-cloud_connector_server +Restart=on-failure +RestartSec=5 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/vizier-kelvin.service b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-kelvin.service new file mode 100644 index 00000000000..7527bf99a60 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-kelvin.service @@ -0,0 +1,26 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=vizier-kelvin — distributed query exec / cross-PEM join (:59300) +After=pl-nats.service vizier-query-broker.service +[Service] +EnvironmentFile=/etc/vizier/middleware.env +# PL_VIZIER_ID must match the cluster id cloud-connector registered. +ExecStart=/usr/local/bin/vizier-kelvin +Restart=on-failure +RestartSec=3 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/vizier-metadata.service b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-metadata.service new file mode 100644 index 00000000000..eda146faa0c --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-metadata.service @@ -0,0 +1,32 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=vizier-metadata — agent registry + (DEGRADED off-k8s) upid->pod map +After=pl-nats.service pl-etcd.service +Requires=pl-etcd.service +Wants=pl-nats.service +[Service] +EnvironmentFile=/etc/vizier/middleware.env +# OFF-K8S CAVEAT: the real metadata service watches the k8s API for pod/service/ +# namespace. There is none on a VM, so upid_to_pod_name/namespace/service resolve +# EMPTY. Supply a static host/cgroup->service map (PL_STATIC_METADATA_FILE) to +# restore readable names at lower fidelity; else the UI shows process/cgroup ids. +Environment=PL_STATIC_METADATA_FILE=/etc/vizier/metadata-static.json +ExecStart=/usr/local/bin/vizier-metadata_server +Restart=on-failure +RestartSec=3 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/middleware/vizier-query-broker.service b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-query-broker.service new file mode 100644 index 00000000000..3fd77563ad4 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/middleware/vizier-query-broker.service @@ -0,0 +1,26 @@ +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +[Unit] +Description=vizier-query-broker — accepts PxL, plans, routes to Kelvin/PEM (:50300) +After=pl-nats.service vizier-metadata.service +Requires=vizier-metadata.service +[Service] +EnvironmentFile=/etc/vizier/middleware.env +ExecStart=/usr/local/bin/vizier-query_broker_server +Restart=on-failure +RestartSec=3 +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/systemd/standalone-pem.service b/src/experimental/dx-standalone-linux/systemd/standalone-pem.service new file mode 100644 index 00000000000..9a3ba8f61e9 --- /dev/null +++ b/src/experimental/dx-standalone-linux/systemd/standalone-pem.service @@ -0,0 +1,25 @@ +[Unit] +Description=Pixie standalone PEM (eBPF capture + ExecuteScript gRPC, no vizier/cloud) +Documentation=file:///.../src/experimental/standalone_pem +After=network-online.target +Wants=network-online.target + +[Service] +# NATIVE binary (bazel runfiles tree extracted to /opt/pixie-pem — see the release +# asset pixie-standalone-pem-*.tar.gz + deploy/INSTALL-VM.md). A native process +# already sees the host /proc + /sys, so there is no mount/host-PID container dance +# and no /sys:rw requirement — Stirling writes /sys/kernel/debug/tracing directly. +Environment=PX_STANDALONE_PEM_PORT=12345 +Environment=PL_TABLE_STORE_DATA_LIMIT_MB=1024 +Environment=PL_HOST_PATH=/ +# /usr/local/bin/standalone-pem is the launcher that execs the runfiles binary. +ExecStart=/usr/local/bin/standalone-pem +AmbientCapabilities=CAP_SYS_ADMIN CAP_BPF CAP_SYS_PTRACE CAP_SYS_RESOURCE CAP_NET_ADMIN CAP_PERFMON CAP_DAC_READ_SEARCH +# resource containment (the k8s DaemonSet's cgroup limits, as systemd): +MemoryMax=2G +CPUQuota=200% +Restart=on-failure +RestartSec=3 + +[Install] +WantedBy=multi-user.target diff --git a/src/experimental/dx-standalone-linux/test/README.md b/src/experimental/dx-standalone-linux/test/README.md new file mode 100644 index 00000000000..5fdfd1437bb --- /dev/null +++ b/src/experimental/dx-standalone-linux/test/README.md @@ -0,0 +1,11 @@ +# Tests — single-VM dx + +- `smoke.sh` — boot dx (referral-only; no eBPF), inject a known attack, assert a + malignant verdict. Runs anywhere. `DX_BIN=/path/to/dx-daemon ./smoke.sh`. +- `nfr.sh` — drive N referrals, report time-to-verdict p50/p95, throughput, + drops, cache hit-rate, RSS/CPU from /metrics. `N=500 ./nfr.sh` against a running dx. + +Pass bars (rebaseline on the target VM): smoke exits 0 (a ruled_in verdict); +nfr p95 time-to-verdict <= 0.5s at the pinned CPU, 0 drops, and — when a real PEM +is attached — bench_unavailable == 0 (else the run is referral-only, stated, not +a silent pass). diff --git a/src/experimental/dx-standalone-linux/test/nfr.sh b/src/experimental/dx-standalone-linux/test/nfr.sh new file mode 100755 index 00000000000..d1ac49469dd --- /dev/null +++ b/src/experimental/dx-standalone-linux/test/nfr.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# nfr.sh — single-VM NFR harness. Drives N referrals through dx and reports the +# pure-Linux equivalent of the k8s report's G1/G2: time-to-verdict p50/p95, +# throughput, backpressure drops, and dx RSS/CPU — all from /metrics. The PEM +# eBPF pull is exercised when a PEM is present (pxdirect), else referral-only. +# +# Env: N (default 500), RATE_SLEEP (default 0 = as fast as possible), +# METRICS (default http://127.0.0.1:9095/metrics), +# DX (default http://127.0.0.1:9099) +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +INJECT="$HERE/../inject/inject-referral.sh" +N="${N:-500}" +DX="${DX:-http://127.0.0.1:9099}" +METRICS="${METRICS:-http://127.0.0.1:9095/metrics}" + +metric() { curl -s "$METRICS" 2>/dev/null | awk -v k="$1" '$1==k {print $2; exit}'; } +hist_q() { # name quantile → interpolate from cumulative _bucket + curl -s "$METRICS" 2>/dev/null | python3 -c ' +import sys,re +name,q=sys.argv[1],float(sys.argv[2]) +b=[];cnt=None +for ln in sys.stdin: + m=re.match(re.escape(name)+r"_bucket\{le=\"([^\"]+)\"\}\s+(\S+)",ln) + if m: b.append((float("inf") if m.group(1)=="+Inf" else float(m.group(1)),float(m.group(2)))) + elif ln.startswith(name+"_count"): cnt=float(ln.split()[-1]) +if not b or not cnt: print("n/a"); sys.exit() +b.sort(); t=q*cnt +for le,c in b: + if c>=t: print(le); break +' "$1" "$2"; } + +echo "[nfr] baseline metrics" +cpu0="$(metric process_cpu_seconds_total)"; t0="$(date +%s)" +verd0="$(metric dx_verdicts_total || echo 0)" + +echo "[nfr] driving $N referrals into $DX ..." +for i in $(seq 1 "$N"); do + case $((i % 4)) in + 0) "$INJECT" benign "$DX" >/dev/null 2>&1 ;; + 1) "$INJECT" argocd-render "$DX" >/dev/null 2>&1 ;; + 2) "$INJECT" log4shell-spawn "$DX" >/dev/null 2>&1 ;; + 3) "$INJECT" cred-escalation "$DX" >/dev/null 2>&1 ;; + esac + [ "${RATE_SLEEP:-0}" != "0" ] && sleep "$RATE_SLEEP" +done +sleep 3 # let the async workups drain + +t1="$(date +%s)"; cpu1="$(metric process_cpu_seconds_total)" +elapsed=$(( t1 - t0 )); [ "$elapsed" -lt 1 ] && elapsed=1 + +echo "" +echo "══════════ single-VM NFR report ══════════" +printf " referrals driven : %s over %ss (%s/s)\n" "$N" "$elapsed" "$(( N / elapsed ))" +printf " time-to-verdict p50 / p95 : %s / %s s\n" "$(hist_q dx_time_to_verdict_seconds 0.50)" "$(hist_q dx_time_to_verdict_seconds 0.95)" +printf " bench-query p95 (PEM pull): %s s (n/a = referral-only, no PEM)\n" "$(hist_q dx_bench_query_duration_seconds 0.95)" +printf " verdicts total : %s\n" "$(metric dx_verdicts_total || echo 0)" +printf " referrals dropped (backp.): %s\n" "$(metric dx_referrals_dropped_total || echo 0)" +printf " bench errors / unavailable: %s / %s\n" "$(metric dx_bench_errors_total || echo 0)" "$(metric dx_bench_unavailable || echo 0)" +printf " cache hit-rate : "; curl -s "$METRICS" 2>/dev/null | python3 -c ' +import sys,re +h=t=b=0.0 +for ln in sys.stdin: + m=re.match(r"dx_bench_pull_total\{[^}]*result=\"([^\"]+)\"[^}]*\}\s+(\S+)",ln) + if m: + v=float(m.group(2)); t+=v + if m.group(1) in ("querycache_hit","telemetry_hit"): h+=v +print(f"{h/t:.2f}" if t else "n/a")' +printf " dx RSS : %s bytes\n" "$(metric process_resident_memory_bytes || echo n/a)" +printf " dx CPU (total / per-verdict): %s / %.4f core-s\n" \ + "$cpu1" "$(python3 -c "v=${verd0:-0}; vv=$(metric dx_verdicts_total || echo 0); c0=${cpu0:-0}; c1=${cpu1:-0}; d=vv-v; print((c1-c0)/d if d>0 else 0)")" +echo "═══════════════════════════════════════════" +echo "[nfr] NFR bar (rebaseline on the target VM): p95 time-to-verdict <= 0.5s @ the pinned CPU; drops == 0; bench_unavailable == 0 when a PEM is attached." diff --git a/src/experimental/dx-standalone-linux/test/smoke.sh b/src/experimental/dx-standalone-linux/test/smoke.sh new file mode 100755 index 00000000000..483a4f575fd --- /dev/null +++ b/src/experimental/dx-standalone-linux/test/smoke.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# smoke.sh — single-VM detection smoke test. Boots dx (referral-only, no eBPF +# needed — event-driven referral evidence, README §3), injects a known attack, +# and asserts a malignant verdict + the exported metric. Runs on any Linux. +# +# Env: DX_BIN (path to the dx-daemon binary). If unset, assumes dx already +# listens on :9099/:9095 (e.g. the systemd/compose deployment). +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +INJECT="$HERE/../inject/inject-referral.sh" +DX_PORT=9099 +METRICS_PORT=9095 +LOG="$(mktemp)" +PID="" + +# shellcheck disable=SC2317 # invoked via trap +cleanup() { [ -n "$PID" ] && kill "$PID" 2>/dev/null; rm -f "$LOG"; } +trap cleanup EXIT + +if [ -n "${DX_BIN:-}" ]; then + echo "[smoke] starting dx: $DX_BIN" + # DX_BENCH=pxdirect with no PEM falls back safely; verdicts still come from the + # event-driven referral seed. DX_SBOB default baseline is fine for the PoC pod. + DX_PORT=$DX_PORT DX_METRICS_PORT=$METRICS_PORT DX_BENCH=pxdirect PX_DIRECT_ADDR=127.0.0.1:12345 \ + "$DX_BIN" >"$LOG" 2>&1 & + PID=$! + for _ in $(seq 1 30); do + curl -sf "http://127.0.0.1:$DX_PORT/healthz" >/dev/null 2>&1 && break + sleep 0.3 + done +fi + +echo "[smoke] catalog:"; grep -m1 'catalog loaded' "$LOG" 2>/dev/null || true + +echo "[smoke] injecting argocd-render (R0001 spawn + R0010 sensitive-file)" +"$INJECT" argocd-render "http://127.0.0.1:$DX_PORT" + +# poll for the DETECTION signal. On a VM without a PEM the bench is blind (no +# network-evidence pull), so a catalog SIGNATURE (ruled_in) may not complete — but +# the event-driven referral evidence (R0001 spawn → invasion) drives the GENERIC +# verdict MALIGNANT, which is the pure-Linux detection. Accept either. +rc=1 +for _ in $(seq 1 40); do + sleep 0.5 + if grep -qE 'generic=MALIGNANT|verdict .*ruled_in|RULE IN' "$LOG" 2>/dev/null; then rc=0; break; fi +done + +if [ "$rc" -eq 0 ]; then + echo "[smoke] PASS — malignant detection on the single VM (no cloud, no k8s):" + grep -m3 -E 'generic=MALIGNANT|verdict .*ruled_in' "$LOG" 2>/dev/null | sed 's/^/ /' || true + if grep -q 'bench UNAVAILABLE' "$LOG" 2>/dev/null; then + echo " NOTE: bench blind (no PEM attached) — generic detection from event-driven evidence." + echo " Attach a standalone_pem for the network-evidence pull + catalog rule-in." + fi +else + echo "[smoke] FAIL — no malignant detection within the window" + tail -20 "$LOG" 2>/dev/null | sed 's/^/ /' +fi +exit $rc diff --git a/src/experimental/dx-standalone-linux/test/verify-assets.sh b/src/experimental/dx-standalone-linux/test/verify-assets.sh new file mode 100755 index 00000000000..b8a3558eb1c --- /dev/null +++ b/src/experimental/dx-standalone-linux/test/verify-assets.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Copyright 2018- The Pixie Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# +# verify-assets.sh — regression guard for the native-systemd deployment defects +# reported in entlein/dx#119: +# #1 PEM tarball top symlink dangled (absolute /app/... build-container target) +# #2 systemd .env inline comment swallowed into the value +# #4 packaged-headers bundle missing (socket_tracer not instantiated → blind) +# +# CI mode (no args): validates the in-repo unit env file (#2) + the unit ExecStart. +# Asset mode: `verify-assets.sh []` validates a +# DOWNLOADED release bundle (#1 symlink resolves, #4 headers present). +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +SYSD="$HERE/../systemd" +fail=0 +ok(){ echo " PASS $1"; } +no(){ echo " FAIL $1"; fail=1; } + +echo "[verify] #2 — no inline comments in dx-standalone.env (systemd keeps them in the value)" +if grep -qE '=[^#]*[[:space:]]+#' "$SYSD/dx-standalone.env"; then + no "dx-standalone.env has a value with a trailing inline comment" + grep -nE '=[^#]*[[:space:]]+#' "$SYSD/dx-standalone.env" | sed 's/^/ /' +else + ok "dx-standalone.env: every comment on its own line" +fi + +echo "[verify] unit ExecStart points at the launcher (native binary reality)" +grep -q 'ExecStart=/usr/local/bin/standalone-pem' "$SYSD/standalone-pem.service" \ + && ok "standalone-pem.service execs the launcher" \ + || no "standalone-pem.service ExecStart not the launcher" + +# Asset mode — validate a downloaded/extracted bundle. +PEMDIR="${1:-}"; PXDIR="${2:-}" +if [ -n "$PEMDIR" ]; then + echo "[verify] #1 — PEM tarball top symlink resolves (not the /app build-container path)" + link="$PEMDIR/standalone_pem/standalone_pem" + if [ -e "$link" ]; then + tgt="$(readlink "$link" 2>/dev/null || true)" + case "$tgt" in + /*) no "top symlink is ABSOLUTE ($tgt) — dangles off the build host" ;; + *) ok "top symlink is relative + resolves ($tgt)" ;; + esac + else + no "top symlink dangles or missing: $link" + fi +fi +if [ -n "$PXDIR" ]; then + echo "[verify] #4 — packaged headers present for socket_tracer" + n=$(ls "$PXDIR"/linux-headers-x86_64-*.tar.gz 2>/dev/null | wc -l) + [ "$n" -gt 0 ] && ok "$n packaged-header tarballs at $PXDIR" || no "no /px/linux-headers-x86_64-*.tar.gz" +fi + +echo "[verify] $([ $fail -eq 0 ] && echo ALL PASS || echo FAILURES)" +exit $fail diff --git a/tools/private/copybara/copy.bara.sky b/tools/private/copybara/copy.bara.sky index ad402fcc930..da58e92c295 100644 --- a/tools/private/copybara/copy.bara.sky +++ b/tools/private/copybara/copy.bara.sky @@ -120,6 +120,8 @@ ignored_dirs = [ # Files/dirs that exist only in the fork and must not be deleted by copybara. fork_only_files = [ + # dx active-diagnosis PoC — pure-Linux standalone PEM-direct showcase (not upstream). + "src/experimental/dx-standalone-linux/**", ".github/workflows/copybara_pixie_oss.yaml", ".github/workflows/perf_clickhouse.yaml", ".github/workflows/perf_soc_attack.yaml",