fix: enable OTel Cloud Logging and Cloud Trace export#112
Open
mohammadp1001 wants to merge 9 commits into
Open
fix: enable OTel Cloud Logging and Cloud Trace export#112mohammadp1001 wants to merge 9 commits into
mohammadp1001 wants to merge 9 commits into
Conversation
setup_otel() calls get_gcp_exporters(enable_cloud_logging=True) unconditionally at every CLI startup, but pyproject.toml only had the trace and monitoring GCP exporters (added in #110) — not the logging one, so every startup crashed with "No module named 'opentelemetry.exporter.cloud_logging'" and traces were silently never exported. Verified end-to-end against a real alphoryn run: setup_otel() now completes cleanly, and OTel GenAI content logs (gen_ai.system.message, gen_ai.user.message, gen_ai.choice) land in Cloud Logging with timestamps matching the run. Cloud Trace spans did not appear after polling for ~2 minutes post-run; documented as a likely separate, unrelated gap in TELEMETRY_ACCESS.md rather than blocking this fix, since the crash this closes is specifically in the logging exporter path. Closes #111
Three stacked gaps kept spans from ever reaching Cloud Trace: - enable_cloud_tracing was never passed to get_gcp_exporters(), so no span exporter was constructed - the OTLP span exporter needs opentelemetry-exporter-otlp-proto-http, which wasn't a declared dependency - telemetry.googleapis.com rejects span batches whose OTel Resource is missing gcp.project_id; ADK's default resource detector only reads it from OTEL_RESOURCE_ATTRIBUTES, so it's now set from google.auth.default() before providers are created Verified end-to-end: a real alphoryn run (run-8/session-0001) now produces a 22-span trace visible via trace_v1.ListTraces.
Cloud Trace span export is now verified working end-to-end (see otel.py fix), so the doc's "unconfirmed"/follow-up language is replaced with the actual verification results.
mohammadp1001
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
alphoryn/telemetry/otel.py:setup_otel()is called unconditionally at every CLI startup and is meant to wire up both Cloud Logging and Cloud Trace export by default. Three stacked bugs kept it from actually working end-to-end:No module named 'opentelemetry.exporter.cloud_logging'. PR fix: add OTel GCP exporter packages to enable Cloud Trace export #110 addedopentelemetry-exporter-gcp-traceandopentelemetry-exporter-gcp-monitoringtopyproject.tomlbut missedopentelemetry-exporter-gcp-logging, the package that actually provides that module.setup_otel()only passedenable_cloud_logging=Truetoget_gcp_exporters()—enable_cloud_tracingdefaults toFalse, so the span exporter/processor was never constructed at all.telemetry.googleapis.comrejects any span batch whose OTelResourcelacks agcp.project_idattribute. ADK's resource detector only reads it from the standardOTEL_RESOURCE_ATTRIBUTESenv var, sosetup_otel()now resolves the project ID viagoogle.auth.default()and sets that env var before providers are built. This also required addingopentelemetry-exporter-otlp-proto-httpas a dependency (the actual span exporter ADK uses), pinned to matchgoogle-adk'sopentelemetry-api/opentelemetry-sdkversion constraints.Verification
Ran a real
alphoryn runsession end-to-end (run-8/session-0001, tickers SPY/QQQ) against thealphorynGCP project:setup_otel()completes with no warning (previously crashed every time)SESSION_START→TOOL_CALL→SIGNAL_SNAPSHOT_BUILT→AGENT_DECISION→SESSION_END) confirmed landing in Cloud Logging underlogName="projects/alphoryn/logs/alphoryn"gen_ai.system.message,gen_ai.user.message,gen_ai.choice) confirmed landing in Cloud Logging, timestamps matching the run exactlytrace_v1.ListTraces()— a 22-span trace for the run appeared (invocation→invoke_agent alphoryn_main_agent→call_llm→generate_content gemini-2.5-pro→execute_tool ...), matching the run's actual tool-call sequenceAlso discovered and documented (not fixed here, out of scope):
python -m alphoryn.cli.main run ...silently no-ops on this dev machine (exit 0, no output, no DB record) — thepython -c "from alphoryn.cli.main import app; ...; app()"invocation style is confirmed working and is what was used for verification above.Test plan
pip install "opentelemetry-exporter-gcp-logging>=1.12.0a0" "opentelemetry-exporter-otlp-proto-http>=1.36.0"thensetup_otel()completes without warningalphoryn runfor one session; confirmedSESSION_START...SESSION_ENDevent chain in Cloud Logginggen_ai.*) present in Cloud Logging with matching timestampstrace_v1.ListTraces()for the same runCloses #111