feat: opt-in OpenTelemetry tracing

[rahulcrl]

Summary

Opt-in OpenTelemetry tracing. Off by default. Enable via CRDB_MCP_OTEL_FILE or OTEL_EXPORTER_OTLP_ENDPOINT (Datadog Agent, OTel Collector).

What it produces

• Root span per MCP tool call (middleware.ToolCallSpan).
• Child span per SQL statement with db.system.name, db.query.text,
  db.operation.name, db.rows_affected / db.response.returned_rows.
• db.query.text is redacted via cockroachdb-parser
  (FmtAnonymize + FmtHideConstants), same pattern as managed-service's
  pkg/mcpserver/handler.go. No PII in spans.
• Zap records also mirror to the OTel log pipeline via otelzap.
  Trace-context propagation onto log records is a follow-up.

Follow up

• Trace-context propagation onto log records (feat: log-trace correlation via ctx-bound logger #35).

Test plan

• go build, go vet, go test ./... clean
• Redaction verified end-to-end: PII literal sent via select_query
  landed in Jaeger as _
• All 13 tools exercised against a live cluster; parent/child span
  structure visible in Jaeger

[arjunmahishi]

This is an AI assisted code review.

Overall this is a well-scoped, high-quality PR: opt-in with zero cost when off, serious PII handling (SQL redaction plus pgx-error scrubbing), and it follows the idiomatic OTel-Go scaffolding (Resource/provider/shutdown, DB semconv, correct SpanKinds). Two non-blocking nits below.

[arjunmahishi]

nit (non-blocking): The tracer scope name here duplicates service.name:

const tracerName = "cockroachdb-mcp-server"
func Tracer() trace.Tracer { return otelapi.Tracer(tracerName) }

The name passed to otel.Tracer() sets the span's instrumentation scope (otel.scope.name), which is meant to identify what code produced the span — deliberately distinct from service.name (already set on the Resource to cockroachdb-mcp-server). Convention is to use the Go import path, e.g. github.com/cockroachdb/cockroachdb-mcp-server/otel.

As-is, scope and service name are identical, so every span carries the same scope and we lose the ability to filter/group spans by originating component. Since spans are actually created from two packages (middleware for the tool span, db for SQL spans), the most idiomatic fix would be two package-named tracers rather than one shared service-named one — so tool spans and SQL spans get distinct scopes. Fine to punt, but flagging so we match the OTel idiom and keep scope useful as a grouping dimension.

[rahulcrl]

Done. Added the shared service-named Tracer() helper; middleware and db now pass their Go import paths as the scope, so tool spans and SQL spans carry distinct otel.scope.name values and scope stays useful as a grouping dimension.

[arjunmahishi]

nit / follow-up (non-blocking): The existing ToolCallLogger already attaches duration_ms to the tool-call log, which is fine and worth keeping. But it'd be valuable to also emit tool-call latency as an OTel histogram metric, so MCP performance can be measured in the standard way — percentile/rate dashboards and alerts straight off the metrics pipeline, without scraping logs or reconstructing from spans.

This is the one signal the PR doesn't cover (traces + logs, no metrics). Concretely: a float64 histogram (e.g. mcp.tool.call.duration, unit s) recorded in the tool-call middleware, keyed by tool name and outcome. It also sidesteps the duration_ms sub-millisecond truncation-to-0 issue for free. Doesn't need to land in this PR — happy for it to be a follow-up like the trace-context work.

[arjunmahishi]

Logging on every tool call is too verbose for some operators, and when shipping to an obs platform they may drop tool-call logs entirely or set a low sample rate. A histogram metric is the one signal whose fidelity doesn't degrade under that — it lets them measure MCP performance (p50/p95/p99, throughput, error rate) in a standard way even with logs sampled or off. That's why I'd treat this as genuinely useful rather than just a nit, though still fine as a follow-up.

[rahulcrl]

Agreed on adding the metrics.
Added a stacked PR: 38, which adds mcp.tool.call.duration (unit s), recorded in the tool-call middleware and keyed by tool name and outcome (success, error, tool_error).
Kept it out of this PR to avoid growing the diff further.

[rahulcrl]

Thanks for the review @arjunmahishi!