chore: add otel observability#35
Open
pratapalakshmi wants to merge 26 commits into
Open
Conversation
Add a cluster-internal self-hosted draw.io as an optional Kustomize component (Deployment + ClusterIP Service, image jgraph/drawio:30.0.4) and enable it in the default, managed, and phoenix overlays. - Reject plane-draw-io-wl from each overlay's APP_VERSION image-tag replacement so the pinned third-party tag is not clobbered. - In phoenix, order draw-io after nonroot-security-context and custom-ca (it has no envFrom, like iframely) and reject it from the initContainers image replacement since it has no init container. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Source the draw-io image tag from overlay-vars.DRAWIO_VERSION via a replacement in the component (mirrors how iframely reads overlay-vars), so version bumps are a one-line change in each overlay's vars.yaml instead of editing the component deployment.yaml. The in-file tag is now just a placeholder default. Document the new key in each overlay's vars.yaml.example (the live vars.yaml is gitignored / user-provided). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the otel-observability component to also inject the shared plane-otel-vars ConfigMap into plane-silo-wl, plus an inline OTEL_SERVICE_NAME=silo that overrides the ConfigMap's api value (inline env wins over envFrom). So silo emits service.name=silo (api stays api) sharing the same endpoint + k8s.namespace.name resource attrs. Also documents the OTEL block in vars.yaml.example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ive/space) Extend the otel-observability component to also inject the shared plane-otel-vars ConfigMap into the env-driven OTel consumers: - live (Node startOtel) -> inline OTEL_SERVICE_NAME=live - space (React Router SSR initServerTelemetry) -> inline OTEL_SERVICE_NAME=space Same pattern as silo: shared endpoint/resource-attrs from the ConfigMap, inline service name override per workload. web and admin are intentionally NOT wired: their only OTel is browser/client tracing, configured from the Plane instance config (is_otel_enabled, otlp_endpoint), not deployment env. OTEL_* env on them would be a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… the API Browser/client tracing for web/admin/space is driven by the API: it serves is_otel_enabled/otlp_endpoint/otlp_headers to browsers from FRONTEND_OTEL_ENABLED / FRONTEND_OTLP_ENDPOINT / FRONTEND_OTLP_HEADERS (DB config or API env). So these only need to be set on the API, which already consumes plane-otel-vars. Adds the three keys to the component ConfigMap, vars.yaml(.example) values, and phoenix replacements. Note: FRONTEND_OTLP_ENDPOINT is sent to browsers (public) and must be an internet-reachable, CORS-enabled OTLP/HTTP collector route — not the backend gRPC path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ll coverage)
Extends the component to every OTel-capable Plane workload, each with an inline
OTEL_SERVICE_NAME matching its container (for promtail service_name correlation):
Django (backend-commercial, OTel via plane.celery import):
worker, beat-worker, automation-consumer, outbox-poller, external-api,
webhook-consumer, worker-importers
pi (plane-pi-commercial, logs-only OTel via pi.celery_app.configure_otel):
pi-api, pi-beat, pi-worker
external-api + worker-importers already carry an inline env (GUNICORN_WORKERS),
so their patches append to /env/- instead of creating /env (no clobber).
web/admin stay excluded (browser tracing via the API's FRONTEND_OTEL_* keys).
flux/email/node-runner/mcp images have no OTel bootstrap -> not wired.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… works cross-origin The OTLP web exporter uses navigator.sendBeacon when no headers are set, and sendBeacon always sends credentials (mode 'include') — which the CORS spec rejects against the collector's wildcard Access-Control-Allow-Origin: * (observed: "blocked by CORS ... must not be the wildcard '*' when credentials mode is include"). Setting a non-empty FRONTEND_OTLP_HEADERS forces the exporter onto XHR, which sends no credentials, so the wildcard is accepted. Value is arbitrary; the collector allows all headers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
Added: full OTel coverage + browser tracing (commits
|
….0 in phoenix) The app's configure_otel defaults to parentbased_traceidratio @ 0.1 (10%), so 90% of requests are never exported to Tempo (their logs show trace_flags=0 and the trace_id is unfindable). Expose OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG via the component ConfigMap + vars replacements; phoenix sets ARG=1.0 so every request is traceable in this load/debug env. Lower for high-traffic prod. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sampler Two trace-visibility fixes found while validating tracing on aurtest: - Sampler: parentbased_traceidratio defers to an incoming traceparent's sampled flag and IGNORES OTEL_TRACES_SAMPLER_ARG. With FRONTEND_OTEL_ENABLED=1 the browser/proxy sends a traceparent; if it didn't sample (flag=0), the API span is dropped even at ARG=1.0 — so only root GET traffic showed up, no browser POST/user-action traces. Switch the phoenix overlay to always_on (captures every span, ignores upstream decisions). Documented both the ratio and parent-based gotchas in the component configmap; prod should use parentbased_traceidratio + a ratio with consistent browser sampling. - Resource attribute: use deployment.environment.name (current semconv key, matches the node/pi/browser services) instead of the legacy deployment.environment, which lands under a separate attribute and breaks cross-service environment filtering in Tempo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…CE_ATTRIBUTES
Spans from plane-ns carried namespace but no k8s.cluster.name, so traces weren't
filterable by cluster in Tempo (e.g. {resource.k8s.cluster.name="plane-eks-dev"}
returned nothing). Add k8s.cluster.name to the example so it's stamped on every
span, consistent with the metrics/logs cluster label. (Live vars.yaml — gitignored
— set to plane-eks-dev for this overlay.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ability # Conflicts: # kustomize/components/draw-io/deployment.yaml # kustomize/overlays/default/kustomization.yaml # kustomize/overlays/default/vars.yaml.example # kustomize/overlays/managed/kustomization.yaml # kustomize/overlays/managed/vars.yaml.example # kustomize/overlays/phoenix/kustomization.yaml # kustomize/overlays/phoenix/vars.yaml.example
…stomize Was "loadtest" (stale); this overlay is the kustomize-deployed instance, so label its environment accordingly. Applies to all services (they share the api's OTEL_RESOURCE_ATTRIBUTES via the plane-otel-vars ConfigMap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add replica overrides in base kustomization for plane-api-wl and plane-worker-wl instead of editing base manifests directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pratapalakshmi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Type of Change
Screenshots and Media (if applicable)
Test Scenarios
References