WIP: feat(observability): add base OpenTelemetry span enricher interceptor#17528
WIP: feat(observability): add base OpenTelemetry span enricher interceptor#17528chalmerlowe wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry tracing observability features to google-api-core, including environment variable and client options resolution helpers, and a gRPC client interceptor (OtelSpanEnricher) to enrich active spans with static and dynamic attributes. The feedback suggests using getattr instead of direct __dict__ access on client_options to properly support properties and slotted attributes, and checking static attributes for None values before setting them on the span to prevent potential OpenTelemetry issues.
88915d4 to
1a59011
Compare
Refactored OtelSpanEnricher to OtelUnaryClientInterceptor to act as a span creator (Pure API approach). Integrated feature gating helpers to control tracing. Added dynamic attribute extraction from gRPC metadata. Cleaned up dead code and added samples.
Removed Translate samples as we are shifting focus to Secret Manager and avoiding auto-instrumentation. Fixed a TypeError in pytest.skip usage in test_tracing.py.
Added tests for __init__.py import failures and expanded metadata parsing tests in test_tracing.py to cover all branches and error handling.
c1c5edd to
98b4f27
Compare
|
@gemini-code-assist |
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry tracing support to google-api-core by adding a new OtelUnaryClientInterceptor gRPC client interceptor and its corresponding unit tests. The reviewer identified several issues in the metadata extraction logic of the interceptor, including a potential crash if metadata is null, unhandled bytes values, missing URL-decoding for resource IDs, and overly broad exception handling. A detailed code suggestion was provided to address these issues.
| for key, value in client_call_details.metadata: | ||
| if key == "x-goog-request-params": | ||
| try: | ||
| # x-goog-request-params is urlencoded string of key=value pairs separated by & | ||
| params = dict( | ||
| p.split("=") for p in value.split("&") if "=" in p | ||
| ) | ||
|
|
||
| # Standard resource identifiers are usually in 'name' or 'parent' | ||
| resource_id = params.get("name") or params.get("parent") | ||
| if resource_id: | ||
| span.set_attribute( | ||
| "gcp.resource.destination.id", resource_id | ||
| ) |
There was a problem hiding this comment.
Issues Identified:
- Potential Crash on Null Metadata:
client_call_details.metadatacan beNoneif no metadata is provided for the gRPC call. Iterating over it directly will raise aTypeErrorand crash the RPC call. - Handling of Bytes Metadata Values: Depending on the gRPC transport and environment, metadata values can be passed as
bytesinstead ofstr. Ifvalueisbytes, calling.split("&")with a string argument will raise aTypeErrorand silently fail open. - URL-Decoding of Resource IDs: The values in
x-goog-request-paramsare URL-encoded. Standard resource identifiers (e.g., containing slashes or special characters) should be URL-decoded usingurllib.parse.unquotebefore being set as the span attribute. - Broad Exception Handling: Avoid broad
except Exception:blocks that silently pass or ignore errors. Instead, log the exception to aid in debugging and prevent masking underlying issues.
Recommendation:
Update the metadata extraction loop to handle None metadata gracefully, decode bytes values to str, URL-decode the extracted resource ID, and log any parsing exceptions instead of silently passing.
| for key, value in client_call_details.metadata: | |
| if key == "x-goog-request-params": | |
| try: | |
| # x-goog-request-params is urlencoded string of key=value pairs separated by & | |
| params = dict( | |
| p.split("=") for p in value.split("&") if "=" in p | |
| ) | |
| # Standard resource identifiers are usually in 'name' or 'parent' | |
| resource_id = params.get("name") or params.get("parent") | |
| if resource_id: | |
| span.set_attribute( | |
| "gcp.resource.destination.id", resource_id | |
| ) | |
| for key, value in (client_call_details.metadata or []): | |
| if key == "x-goog-request-params": | |
| try: | |
| # x-goog-request-params is urlencoded string of key=value pairs separated by & | |
| val_str = value.decode("utf-8") if isinstance(value, bytes) else value | |
| params = dict( | |
| p.split("=") for p in val_str.split("&") if "=" in p | |
| ) | |
| # Standard resource identifiers are usually in 'name' or 'parent' | |
| resource_id = params.get("name") or params.get("parent") | |
| if resource_id: | |
| from urllib.parse import unquote | |
| span.set_attribute( | |
| "gcp.resource.destination.id", unquote(resource_id) | |
| ) | |
| except Exception as e: | |
| import logging | |
| logger = logging.getLogger(__name__) | |
| logger.warning("Failed to parse x-goog-request-params: %s", e) |
References
- Avoid broad except Exception: blocks that silently return None or pass. Instead, log the exception to aid in debugging and prevent masking underlying issues.
Note
THIS IS A WIP
This PR introduces the base OpenTelemetry (OTel) gRPC client interceptor (
OtelSpanEnricher) under thegoogle.api_core.observabilitymodule. This interceptor is a key part of our "Delegate and Enrich" tracing strategy, which relies on standard OTel instrumentation to generate baseline network spans (T4 spans) and enriches them with Google Cloud specific domain attributes.Intent & Motivation
To meet the Client Libraries Observability Product Requirements, our generated libraries need to inject standard domain-specific attributes (like repository name, service name, and destination resource ID) onto generated tracing spans.
Rather than manual implementation of span lifecycle and propagation boilerplate in each client transport, this base interceptor allows client libraries to register a callback (
attribute_extractor) during channel initialization. This callback extracts dynamic telemetry attributes directly from request payloads (e.g. database name in Spanner, queue name in PubSub) and injects them into the active OTel span.Changes Included
OtelSpanEnricherclass, inheriting fromgrpc.UnaryUnaryClientInterceptor, which resolves static attributes and extracts dynamic attributes from request payloads to attach to the active recording span.opentelemetry-apito the dependencies block ofpyproject.tomland mapped its lower bound (opentelemetry-api==1.1.0) in the test constraints.grpcbeing installed.pytestpatterns to verify enrichment behavior on recording/non-recording spans, as well as checking that attribute extraction failures do not crash runtime RPC calls.