feat(recipes): add observability recipes for logging and metrics#36
Open
dishant-gupta-glean wants to merge 1 commit into
Open
feat(recipes): add observability recipes for logging and metrics#36dishant-gupta-glean wants to merge 1 commit into
dishant-gupta-glean wants to merge 1 commit into
Conversation
Author
Cloud Provider Recipes AddedAdded AWS and GCP cloud provider integration recipes on top of existing observability recipes: cloud_providers_aws.py (5 examples)
cloud_providers_gcp.py (6 examples)
These recipes depend on PRs:
All examples are runnable with |
dishant-gupta-glean
requested review from
Copilot,
ishan-chawla-glean and
sarthak-kapoor-glean
There was a problem hiding this comment.
Pull request overview
This PR expands the SDK’s observability surface area by adding structured JSON logging, lifecycle event logging, a pluggable metrics provider interface, and AWS/GCP plugin providers, along with a full set of “recipes” and unit tests to demonstrate and validate usage.
Changes:
- Added structured JSON logging formatters and enhanced
setup_connector_logging()to support structured logging, custom formatters, extra handlers, and pluggableLoggerProviders. - Introduced lifecycle event logging and a
MetricsProviderinterface with an in-memory default provider, plus AWS CloudWatch and GCP Cloud Monitoring/Logging provider implementations. - Added a comprehensive set of
recipes/observability/*examples and corresponding unit tests for observability, metrics, and cloud plugins.
Reviewed changes
Copilot reviewed 36 out of 36 changed files in this pull request and generated 34 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit_tests/plugins/gcp/test_cloud_monitoring.py | Unit tests for GCP Cloud Monitoring metrics provider |
| tests/unit_tests/plugins/gcp/test_cloud_logging.py | Unit tests for GCP Cloud Logging provider |
| tests/unit_tests/plugins/gcp/init.py | Test package marker for GCP plugins |
| tests/unit_tests/plugins/aws/test_cloudwatch_metrics.py | Unit tests for AWS CloudWatch metrics provider |
| tests/unit_tests/plugins/aws/test_cloudwatch_logs.py | Unit tests for AWS CloudWatch logs provider |
| tests/unit_tests/plugins/aws/init.py | Test package marker for AWS plugins |
| tests/unit_tests/plugins/init.py | Test package marker for plugins tests |
| tests/unit_tests/observability/test_setup_logging.py | Tests for setup_connector_logging() incl. provider integration |
| tests/unit_tests/observability/test_providers.py | Tests for MetricsProvider and InMemoryMetricsProvider |
| tests/unit_tests/observability/test_metrics_integration.py | Integration tests for metrics recording via ConnectorObservability |
| tests/unit_tests/observability/test_logger_provider.py | Tests for LoggerProvider and ConsoleLoggerProvider |
| tests/unit_tests/observability/test_lifecycle_logging.py | Tests for lifecycle event logging behavior/fields |
| tests/unit_tests/observability/test_formatters.py | Tests for StructuredFormatter and CompactStructuredFormatter |
| tests/unit_tests/observability/init.py | Test package marker for observability tests |
| src/glean/indexing/plugins/gcp/cloud_monitoring.py | GCP Cloud Monitoring MetricsProvider implementation |
| src/glean/indexing/plugins/gcp/cloud_logging.py | GCP Cloud Logging LoggerProvider implementation |
| src/glean/indexing/plugins/gcp/init.py | GCP plugin exports + optional dependency warning wrapper |
| src/glean/indexing/plugins/aws/cloudwatch_metrics.py | AWS CloudWatch MetricsProvider implementation |
| src/glean/indexing/plugins/aws/cloudwatch_logs.py | AWS CloudWatch Logs LoggerProvider implementation |
| src/glean/indexing/plugins/aws/init.py | AWS plugin exports + optional dependency warning wrapper |
| src/glean/indexing/plugins/init.py | Plugins package marker/docstring |
| src/glean/indexing/observability/providers.py | MetricType, MetricsProvider ABC, and in-memory provider |
| src/glean/indexing/observability/observability.py | ConnectorObservability lifecycle logging + metrics provider integration |
| src/glean/indexing/observability/logging.py | LoggerProvider ABC + ConsoleLoggerProvider |
| src/glean/indexing/observability/formatters.py | Structured JSON logging formatters |
| src/glean/indexing/observability/init.py | Public exports for new observability APIs |
| recipes/observability/performance_tracking.py | Performance tracker and progress callback recipes |
| recipes/observability/metrics_recording.py | Built-in metrics recording recipes |
| recipes/observability/metrics_providers.py | Custom metrics provider recipes (Prometheus/DataDog/CloudWatch) |
| recipes/observability/logging_setup.py | Logging configuration recipes |
| recipes/observability/lifecycle_events.py | Crawl lifecycle logging recipes |
| recipes/observability/decorators.py | Decorator-based observability recipes |
| recipes/observability/cloud_providers_gcp.py | GCP integration recipes |
| recipes/observability/cloud_providers_aws.py | AWS integration recipes |
| recipes/observability/init.py | Recipes package marker/docstring |
| pyproject.toml | Adds [aws] and [gcp] optional dependency groups |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+36
to
+37
| obs.log_batch_upload_started(batch_index=0, batch_size=100) | ||
| obs.log_batch_upload_completed(batch_index=0) |
Comment on lines
+99
to
+101
| obs.log_batch_upload_started(batch_index=0, batch_size=50) | ||
| obs.record_batch_upload_latency(350.0) | ||
| obs.log_batch_upload_completed(batch_index=0) |
Comment on lines
+156
to
+159
| except Exception as e: | ||
| obs.fail_execution(e) | ||
|
|
||
| obs.record_crawl_outcome("failed", error_message="API rate limit exceeded") |
Comment on lines
+36
to
+37
| obs.log_batch_upload_started(batch_index=0, batch_size=100) | ||
| obs.log_batch_upload_completed(batch_index=0) |
Comment on lines
+107
to
+109
| obs.log_batch_upload_started(batch_index=0, batch_size=50) | ||
| obs.record_batch_upload_latency(350.0) | ||
| obs.log_batch_upload_completed(batch_index=0) |
Comment on lines
+1
to
+3
| """GCP Cloud Logging and Monitoring integration recipes.""" | ||
|
|
||
| import logging |
| region_name: AWS region name | ||
| create_log_group: Whether to create log group if it doesn't exist | ||
| """ | ||
| import watchtower |
|
|
||
| from unittest.mock import MagicMock, patch | ||
|
|
||
| from glean.indexing.observability import MetricType |
Comment on lines
+3
to
+5
| try: | ||
| from glean.indexing.plugins.aws.cloudwatch_logs import CloudWatchLogsProvider | ||
| from glean.indexing.plugins.aws.cloudwatch_metrics import CloudWatchMetricsProvider |
Comment on lines
+3
to
+5
| try: | ||
| from glean.indexing.plugins.gcp.cloud_logging import CloudLoggingProvider | ||
| from glean.indexing.plugins.gcp.cloud_monitoring import CloudMonitoringProvider |
| @@ -0,0 +1,177 @@ | |||
| """AWS CloudWatch integration recipes.""" | |||
There was a problem hiding this comment.
There are few lint kind of issues present which are followed in the repo and also enforced by the ruff commands. Can you please resolve these?
Bot has commented regarding a few of these below
| self.status_code = response.status_code if response is not None else None | ||
|
|
||
|
|
||
| class PullHttpClient: |
There was a problem hiding this comment.
@dishant-gupta-glean is this rebased properly?
| """Generic source-side HTTP client for connector data clients.""" | ||
|
|
||
| import logging | ||
| import random |
Author
There was a problem hiding this comment.
waiting to get all observability PRs merged then will rebase recipes all toughter
Add example recipes for using the observability SDK components: - Logging setup (structured, plain, custom level) - Metrics recording patterns - Lifecycle event patterns - Decorator patterns for timing/tracking - Performance tracking examples - Cloud provider recipes (AWS CloudWatch + GCP Cloud Logging/Monitoring) - Pull HTTP client recipe Also add recipes/__init__.py and tests/unit_tests/recipes/__init__.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dishant-gupta-glean
force-pushed
the
recipes/observability
branch
from
3e5bccf to
442ce54
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive observability recipes on top of
pull-layer-recipes-pr1showing all logging, metrics, and cloud provider integration capabilities from PRs #32, #33, #34, #37, #38, and #39.Depends on: pull-layer-recipes-pr1
Recipe Files
Core Observability
1. logging_setup.py - Logging Configuration
setup_basic_logging()- Human-readable console loggingsetup_structured_logging()- JSON structured logging for productionsetup_compact_structured_logging()- Compact JSON (omits empty fields)setup_custom_format_logging()- Custom log format stringssetup_multi_handler_logging()- Console + file loggingsetup_custom_structured_fields()- Extra fields in every log2. lifecycle_events.py - Crawl Phase Tracking
track_basic_execution()- Start/end executiontrack_execution_with_context()- With run_id/datasource/crawl_modetrack_failed_execution()- Error handlingtrack_data_fetch_phase()- Data fetching from sourcetrack_transform_phase()- Data transformationtrack_batch_upload_phase()- Batch uploads to Gleantrack_full_crawl_lifecycle()- Complete crawl example3. metrics_providers.py - Custom Provider Implementations
PrometheusMetricsProvider- prometheus_client integrationDataDogMetricsProvider- datadog library integrationCloudWatchMetricsProvider- boto3 CloudWatch integrationuse_in_memory_metrics()- Default provider example4. decorators.py - Automatic Logging Decorators
@with_observability()- Auto-log all public methods@with_observability(exclude_methods=[...])- Exclude specific methods@with_observability(include_args=True, include_return=True)- Verbose logging@track_crawl_progress- Auto-track item counts5. performance_tracking.py - Context Managers
PerformanceTracker- Track operation durationPerformanceTrackerwith observability integrationProgressCallback- Track progress with known/unknown totals6. metrics_recording.py - Built-in Metrics
record_upload_metrics()- Batch size and throughputrecord_api_metrics()- Latency, count, errorsrecord_retry_metrics()- Retry attemptsrecord_crawl_outcome_metrics()- Success/failurerecord_custom_metrics()- Custom counters and timersrecord_with_labels()- Metrics with custom labelsCloud Provider Integration
7. cloud_providers_aws.py - AWS CloudWatch Integration (5 examples)
use_cloudwatch_metrics()- CloudWatch Metrics onlyuse_cloudwatch_logs()- CloudWatch Logs onlyuse_cloudwatch_full_observability()- Metrics + logs togetheruse_cloudwatch_with_custom_buffer()- High-volume metrics bufferinguse_cloudwatch_error_tracking()- Error monitoring and trackingInstallation:
uv add glean-indexing-sdk[aws]8. cloud_providers_gcp.py - GCP Cloud Logging/Monitoring (6 examples)
use_cloud_monitoring()- Cloud Monitoring onlyuse_cloud_logging()- Cloud Logging onlyuse_gcp_full_observability()- Metrics + logs with k8s resourcesuse_cloud_monitoring_with_custom_buffer()- High-volume metrics bufferinguse_gcp_error_tracking()- Error monitoring with generic_task resourcesuse_gce_instance_monitoring()- GCE instance-specific monitoringInstallation:
uv add glean-indexing-sdk[gcp]Recipe Pattern
All recipes follow the established pattern:
if __name__ == "__main__"sectionsCoverage
Covers all observability features from:
Example Usage
AWS CloudWatch
GCP Cloud Logging/Monitoring