Skip to content

perf(spanner): create Cloud Monitoring MetricServiceClient lazily#5

Open
fornwall wants to merge 1 commit into
mainfrom
fix/p1-6-lazy-metric-service-client
Open

perf(spanner): create Cloud Monitoring MetricServiceClient lazily#5
fornwall wants to merge 1 commit into
mainfrom
fix/p1-6-lazy-metric-service-client

Conversation

@fornwall

Copy link
Copy Markdown
Owner

Issue (review P1-6)

SpannerCloudMonitoringExporter.create() eagerly called MetricServiceClient.create(...), which builds a second full GAPIC client — its own gRPC channel, executor and watchdog — dedicated to Cloud Monitoring.

This runs synchronously inside BuiltInMetricsProvider.getOrCreateOpenTelemetry(), which is invoked from the GapicSpannerRpc constructor (getFallbackOpenTelemetrySpannerOptions.getBuiltInOpenTelemetry). So every Spanner client stands up the whole monitoring client stack before the Spanner stub is usable — even though the first metric export only happens after the first reader interval (~1 minute later). Pure wasted startup latency and thread/channel allocation.

I traced the path against source to confirm the finding is real:
GapicSpannerRpc ctor → getFallbackOpenTelemetrySpannerOptions.getBuiltInOpenTelemetryBuiltInMetricsProvider.getOrCreateOpenTelemetrySpannerCloudMonitoringExporter.createMetricServiceClient.create.

Fix

Defer building the MetricServiceClient to the first non-empty export():

  • create() now only builds the (cheap) MetricServiceSettings and hands the exporter a MetricServiceClientFactory that constructs the client on demand.
  • getOrCreateClient() creates it exactly once under a lock. export() returns early for empty collections, so idle reader intervals never create the client.
  • shutdown() is now idempotent via a dedicated shutdown flag and skips work entirely when the client was never created (returns success — nothing to tear down).
  • The heavy work (channel/executor/watchdog) is what's deferred; building settings stays eager, so getMetricServiceClient().getSettings() (used by testUniverseDomain) still works by forcing creation.

Chose the lazy-client approach over "build the OTel stack on a background thread" because it's fully contained within the exporter, avoids new background-thread ordering/visibility concerns, and directly targets the expensive part.

Tests

  • testClientCreatedLazilyOnFirstNonEmptyExport — asserts the client is not created at construction, not created by an empty export, created exactly once on the first non-empty export, and reused thereafter.
  • testShutdownBeforeExportDoesNotCreateClient — asserts shutdown before any export succeeds without creating a client, is idempotent, and a post-shutdown export fails without creating one.

All 13 tests in SpannerCloudMonitoringExporterTest pass. fmt:check clean.

Not addressed (related P2)

The review bundles a P2 note: the gRPC built-in-metrics path (SpannerOptions.enablegRPCMetrics) is not gated on emulator/NoCredentials, so emulator/CI runs still build the OTel meter provider despite never being able to export. That's a separate behavioral change and is left as a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Ci2KFsNgWdHCno4dkXdUU

SpannerCloudMonitoringExporter.create() eagerly called
MetricServiceClient.create(), building a second full GAPIC client
(its own channel, executor and watchdog) for Cloud Monitoring. This ran
synchronously from BuiltInMetricsProvider.getOrCreateOpenTelemetry(),
which is invoked from the GapicSpannerRpc constructor - so every Spanner
client paid the cost of standing up the monitoring client before the
Spanner stub was even usable, even though the first metric export only
happens after the first reader interval (~1 min later).

Defer building the MetricServiceClient to the first non-empty export():
- create() now only builds the (cheap) MetricServiceSettings and hands
  the exporter a MetricServiceClientFactory that constructs the client on
  demand.
- getOrCreateClient() creates it once under a lock; export() returns
  early for empty collections so an idle interval never creates it.
- shutdown() is now idempotent via a dedicated flag and skips work when
  the client was never created.

Adds tests asserting the client is created lazily (once, on first
non-empty export) and that shutting down before any export succeeds
without creating a client.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Ci2KFsNgWdHCno4dkXdUU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant