perf(spanner): create Cloud Monitoring MetricServiceClient lazily#5
Open
fornwall wants to merge 1 commit into
Open
perf(spanner): create Cloud Monitoring MetricServiceClient lazily#5fornwall wants to merge 1 commit into
fornwall wants to merge 1 commit into
Conversation
SpannerCloudMonitoringExporter.create() eagerly called MetricServiceClient.create(), building a second full GAPIC client (its own channel, executor and watchdog) for Cloud Monitoring. This ran synchronously from BuiltInMetricsProvider.getOrCreateOpenTelemetry(), which is invoked from the GapicSpannerRpc constructor - so every Spanner client paid the cost of standing up the monitoring client before the Spanner stub was even usable, even though the first metric export only happens after the first reader interval (~1 min later). Defer building the MetricServiceClient to the first non-empty export(): - create() now only builds the (cheap) MetricServiceSettings and hands the exporter a MetricServiceClientFactory that constructs the client on demand. - getOrCreateClient() creates it once under a lock; export() returns early for empty collections so an idle interval never creates it. - shutdown() is now idempotent via a dedicated flag and skips work when the client was never created. Adds tests asserting the client is created lazily (once, on first non-empty export) and that shutting down before any export succeeds without creating a client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Ci2KFsNgWdHCno4dkXdUU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue (review P1-6)
SpannerCloudMonitoringExporter.create()eagerly calledMetricServiceClient.create(...), which builds a second full GAPIC client — its own gRPC channel, executor and watchdog — dedicated to Cloud Monitoring.This runs synchronously inside
BuiltInMetricsProvider.getOrCreateOpenTelemetry(), which is invoked from theGapicSpannerRpcconstructor (getFallbackOpenTelemetry→SpannerOptions.getBuiltInOpenTelemetry). So every Spanner client stands up the whole monitoring client stack before the Spanner stub is usable — even though the first metric export only happens after the first reader interval (~1 minute later). Pure wasted startup latency and thread/channel allocation.I traced the path against source to confirm the finding is real:
GapicSpannerRpcctor →getFallbackOpenTelemetry→SpannerOptions.getBuiltInOpenTelemetry→BuiltInMetricsProvider.getOrCreateOpenTelemetry→SpannerCloudMonitoringExporter.create→MetricServiceClient.create.Fix
Defer building the
MetricServiceClientto the first non-emptyexport():create()now only builds the (cheap)MetricServiceSettingsand hands the exporter aMetricServiceClientFactorythat constructs the client on demand.getOrCreateClient()creates it exactly once under a lock.export()returns early for empty collections, so idle reader intervals never create the client.shutdown()is now idempotent via a dedicatedshutdownflag and skips work entirely when the client was never created (returns success — nothing to tear down).getMetricServiceClient().getSettings()(used bytestUniverseDomain) still works by forcing creation.Chose the lazy-client approach over "build the OTel stack on a background thread" because it's fully contained within the exporter, avoids new background-thread ordering/visibility concerns, and directly targets the expensive part.
Tests
testClientCreatedLazilyOnFirstNonEmptyExport— asserts the client is not created at construction, not created by an empty export, created exactly once on the first non-empty export, and reused thereafter.testShutdownBeforeExportDoesNotCreateClient— asserts shutdown before any export succeeds without creating a client, is idempotent, and a post-shutdown export fails without creating one.All 13 tests in
SpannerCloudMonitoringExporterTestpass.fmt:checkclean.Not addressed (related P2)
The review bundles a P2 note: the gRPC built-in-metrics path (
SpannerOptions.enablegRPCMetrics) is not gated on emulator/NoCredentials, so emulator/CI runs still build the OTel meter provider despite never being able to export. That's a separate behavioral change and is left as a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_016Ci2KFsNgWdHCno4dkXdUU