Skip to content

WIP: feat(observability): add base OpenTelemetry span enricher interceptor#17528

Draft
chalmerlowe wants to merge 7 commits into
mainfrom
feat/otel-prototype-interceptor
Draft

WIP: feat(observability): add base OpenTelemetry span enricher interceptor#17528
chalmerlowe wants to merge 7 commits into
mainfrom
feat/otel-prototype-interceptor

Conversation

@chalmerlowe

@chalmerlowe chalmerlowe commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Note

THIS IS A WIP

This PR introduces the base OpenTelemetry (OTel) gRPC client interceptor (OtelSpanEnricher) under the google.api_core.observability module. This interceptor is a key part of our "Delegate and Enrich" tracing strategy, which relies on standard OTel instrumentation to generate baseline network spans (T4 spans) and enriches them with Google Cloud specific domain attributes.

Intent & Motivation

To meet the Client Libraries Observability Product Requirements, our generated libraries need to inject standard domain-specific attributes (like repository name, service name, and destination resource ID) onto generated tracing spans.

Rather than manual implementation of span lifecycle and propagation boilerplate in each client transport, this base interceptor allows client libraries to register a callback (attribute_extractor) during channel initialization. This callback extracts dynamic telemetry attributes directly from request payloads (e.g. database name in Spanner, queue name in PubSub) and injects them into the active OTel span.

Changes Included

  • OtelSpanEnricher Interceptor: Added the OtelSpanEnricher class, inheriting from grpc.UnaryUnaryClientInterceptor, which resolves static attributes and extracts dynamic attributes from request payloads to attach to the active recording span.
  • Direct Dependency Integration: Added opentelemetry-api to the dependencies block of pyproject.toml and mapped its lower bound (opentelemetry-api==1.1.0) in the test constraints.
  • Robust Fallbacks & Graceful Degradation: Handled missing gRPC installation environments cleanly by using soft imports in the package initialization, ensuring the library remains completely importable and operational without grpc being installed.
  • Unit Tests: Provided comprehensive unit tests using parametrized pytest patterns to verify enrichment behavior on recording/non-recording spans, as well as checking that attribute extraction failures do not crash runtime RPC calls.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces OpenTelemetry tracing observability features to google-api-core, including environment variable and client options resolution helpers, and a gRPC client interceptor (OtelSpanEnricher) to enrich active spans with static and dynamic attributes. The feedback suggests using getattr instead of direct __dict__ access on client_options to properly support properties and slotted attributes, and checking static attributes for None values before setting them on the span to prevent potential OpenTelemetry issues.

Comment thread packages/google-api-core/google/api_core/observability/options.py Outdated
Comment thread packages/google-api-core/google/api_core/observability/tracing.py Outdated
@chalmerlowe chalmerlowe changed the title feat(observability): add base OpenTelemetry span enricher interceptor WIP: feat(observability): add base OpenTelemetry span enricher interceptor Jun 24, 2026
@chalmerlowe
chalmerlowe force-pushed the feat/otel-prototype-interceptor branch from 88915d4 to 1a59011 Compare July 24, 2026 14:59
Refactored OtelSpanEnricher to OtelUnaryClientInterceptor to act as a span creator (Pure API approach). Integrated feature gating helpers to control tracing. Added dynamic attribute extraction from gRPC metadata. Cleaned up dead code and added samples.
Removed Translate samples as we are shifting focus to Secret Manager and avoiding auto-instrumentation. Fixed a TypeError in pytest.skip usage in test_tracing.py.
Added tests for __init__.py import failures and expanded metadata parsing tests in test_tracing.py to cover all branches and error handling.
@chalmerlowe
chalmerlowe force-pushed the feat/otel-prototype-interceptor branch from c1c5edd to 98b4f27 Compare July 24, 2026 15:57
@chalmerlowe

Copy link
Copy Markdown
Contributor Author

@gemini-code-assist
please review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces OpenTelemetry tracing support to google-api-core by adding a new OtelUnaryClientInterceptor gRPC client interceptor and its corresponding unit tests. The reviewer identified several issues in the metadata extraction logic of the interceptor, including a potential crash if metadata is null, unhandled bytes values, missing URL-decoding for resource IDs, and overly broad exception handling. A detailed code suggestion was provided to address these issues.

Comment on lines +76 to +89
for key, value in client_call_details.metadata:
if key == "x-goog-request-params":
try:
# x-goog-request-params is urlencoded string of key=value pairs separated by &
params = dict(
p.split("=") for p in value.split("&") if "=" in p
)

# Standard resource identifiers are usually in 'name' or 'parent'
resource_id = params.get("name") or params.get("parent")
if resource_id:
span.set_attribute(
"gcp.resource.destination.id", resource_id
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Issues Identified:

  1. Potential Crash on Null Metadata: client_call_details.metadata can be None if no metadata is provided for the gRPC call. Iterating over it directly will raise a TypeError and crash the RPC call.
  2. Handling of Bytes Metadata Values: Depending on the gRPC transport and environment, metadata values can be passed as bytes instead of str. If value is bytes, calling .split("&") with a string argument will raise a TypeError and silently fail open.
  3. URL-Decoding of Resource IDs: The values in x-goog-request-params are URL-encoded. Standard resource identifiers (e.g., containing slashes or special characters) should be URL-decoded using urllib.parse.unquote before being set as the span attribute.
  4. Broad Exception Handling: Avoid broad except Exception: blocks that silently pass or ignore errors. Instead, log the exception to aid in debugging and prevent masking underlying issues.

Recommendation:

Update the metadata extraction loop to handle None metadata gracefully, decode bytes values to str, URL-decode the extracted resource ID, and log any parsing exceptions instead of silently passing.

Suggested change
for key, value in client_call_details.metadata:
if key == "x-goog-request-params":
try:
# x-goog-request-params is urlencoded string of key=value pairs separated by &
params = dict(
p.split("=") for p in value.split("&") if "=" in p
)
# Standard resource identifiers are usually in 'name' or 'parent'
resource_id = params.get("name") or params.get("parent")
if resource_id:
span.set_attribute(
"gcp.resource.destination.id", resource_id
)
for key, value in (client_call_details.metadata or []):
if key == "x-goog-request-params":
try:
# x-goog-request-params is urlencoded string of key=value pairs separated by &
val_str = value.decode("utf-8") if isinstance(value, bytes) else value
params = dict(
p.split("=") for p in val_str.split("&") if "=" in p
)
# Standard resource identifiers are usually in 'name' or 'parent'
resource_id = params.get("name") or params.get("parent")
if resource_id:
from urllib.parse import unquote
span.set_attribute(
"gcp.resource.destination.id", unquote(resource_id)
)
except Exception as e:
import logging
logger = logging.getLogger(__name__)
logger.warning("Failed to parse x-goog-request-params: %s", e)
References
  1. Avoid broad except Exception: blocks that silently return None or pass. Instead, log the exception to aid in debugging and prevent masking underlying issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant