Logging overhaul by sstill88 · Pull Request #137 · SkyTruth/cerulean-cloud

sstill88 · 2025-01-08T23:47:29Z

Description

This updates all of the logs to be structured (json_payload) logs. It also includes a notebook that demonstrates how to query the structured logs.

Changes

Added logs and/or updated logging/print statements to queryable logs
added sigterm handler to cerulean_cloud/cloud_run_orchestrator/handler.py. This prints a log if an event like a timeout or manual stop occurs. Though it cannot print the type of sigterm received, some likely reasons for specific messages are documented in the notebook
updated action.yaml to include the utils file that has structured_log
Added notebooks/CloudRunLogs.ipynb and associated notebooks/logging_utils.py file for querying the logger
Updated test/test_cerulean_cloud/test_roda_sentinelhub_client.py to include method="GET" (this test wasn't catching errors because it didn't have the right method)

Reviewers

Does it look reasonable?
Test out notebooks/CloudRunLogs.ipynb:
- Make sure this version is deployed on TEST
- Run a list of scenes
- Set the correct revision_name in the notebook
- Run the cells of the notebook
Do we like the logging format?

github-actions · 2025-01-08T23:52:10Z

🍹 `preview` on cerulean-cloud-images/test

Pulumi report

Previewing update (test):
@ previewing update..........

@ previewing update..............
pulumi:pulumi:Stack cerulean-cloud-images-test running
@ previewing update....
gcp:container:Registry cerulean-cloud-images-test-registry
@ previewing update.....
docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image Building your image for linux/amd64 architecture.
docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image Building your image for linux/amd64 architecture.
docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image Building your image for linux/amd64 architecture.
docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image Building your image for linux/amd64 architecture.
@ previewing update....
docker:index:Image cerulean-cloud-images-test-cr-tipg-image Building your image for linux/amd64 architecture.
docker:index:Image cerulean-cloud-images-test-cr-tipg-image Building your image for linux/amd64 architecture.
pulumi:pulumi:Stack cerulean-cloud-images-test
Diagnostics:
docker:index:Image (cerulean-cloud-images-test-cr-orchestrator-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

docker:index:Image (cerulean-cloud-images-test-cr-offset-tile-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

docker:index:Image (cerulean-cloud-images-test-cr-tipg-image):
Building your image for linux/amd64 architecture.
To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

Resources:
5 unchanged

github-actions · 2025-01-08T23:54:21Z

🍹 `preview` on cerulean-cloud/test

Pulumi report

   Previewing update (test):

@ previewing update.....
   pulumi:pulumi:Stack cerulean-cloud-test running 
@ previewing update....
   pulumi:providers:docker cerulean-cloud-images-test-gcr  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-ais  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-tasks-ais-analysis  
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-ais  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais create replacement [diff: ~detectMd5hash,name,source]
   gcp:sql:User cerulean-cloud-test-database-users  
   gcp:projects:IAMMember cerulean-cloud-test-cf-ais-iam  
   gcp:sql:Database cerulean-cloud-test-database  
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-ais update [diff: ~environmentVariables,secretEnvironmentVariables,sourceArchiveObject]
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-ais-invoker  
@ previewing update..........
   gcp:serviceaccount:Account cerulean-cloud-test-cr-offset-tile  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-offset  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator  
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-secretmanagerSecretAccessor  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-offset-tile-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-offset-tiles update [diff: ~metadata,template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-offset  
@ previewing update.....
   aws:iam:Role cerulean-cloud-test-lambda-titiler-role  
   aws:s3:Bucket cerulean-cloud-test-titiler-lambda-archive  
   aws:iam:Policy cerulean-cloud-test-lambda-titiler-policy  
   aws:sns:Topic cerulean-cloud-test-lambda-APIAbuseAlert  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-sr  
   aws:apigatewayv2:Api cerulean-cloud-test-lambda-titiler-api  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cr-orchestrator  
@ previewing update....
   gcp:serviceaccount:Account cerulean-cloud-test-cr-orchestrator  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-sr  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-tipg  
   aws:iam:Role cerulean-cloud-test-lambda-sentinel1-iam  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment2  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-support  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jona  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jason  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-aemon  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr create replacement [diff: ~detectMd5hash,name,source]
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudTasksEnqueuer  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run create replacement [diff: ~detectMd5hash,name,source]
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-secretmanagerSecretAccessor  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-sentinel1-basic-execution  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cf-sr-iam  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-tipg-secret-accessor-binding  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-secretmanagerSecretAccessor  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-orchestrator-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-tipg update [diff: ~metadata]
~  gcp:cloudrun:Service cerulean-cloud-test-cr-orchestrator update [diff: ~metadata,template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-tipg  
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-orchestrator  
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-sr update [diff: ~secretEnvironmentVariables,sourceArchiveObject]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-historical-run update [diff: ~secretEnvironmentVariables,sourceArchiveObject]
   aws:lambda:Function cerulean-cloud-test-lambda-sentinel1-sub  
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-sr-invoker  
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-historical-run-invoker  
   aws:lambda:Permission cerulean-cloud-test-lambda-sentinel1-permission  
   aws:sns:TopicSubscription cerulean-cloud-test-sentinel1-subscription  
@ previewing update..........................................................................................................................
~  aws:s3:BucketObject cerulean-cloud-test-titiler-lambda-archive update [diff: ~source]
~  aws:lambda:Function cerulean-cloud-test-lambda-titiler-sentinel update [diff: ~sourceCodeHash]
   aws:apigatewayv2:Integration cerulean-cloud-test-lambda-titiler-integration  
   aws:cloudwatch:MetricAlarm cerulean-cloud-test-lambda-titiler-alarm  
   aws:lambda:Permission cerulean-cloud-test-lambda-titiler-permission  
   aws:apigatewayv2:Route cerulean-cloud-test-lambda-titiler-route  
   aws:apigatewayv2:Stage cerulean-cloud-test-lambda-titiler-stage  
   pulumi:pulumi:Stack cerulean-cloud-test running Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   pulumi:pulumi:Stack cerulean-cloud-test running Checking Docker is available...
   pulumi:pulumi:Stack cerulean-cloud-test running Building container image...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully built container image with id sha256:285566eca62f43a437d6ad9970241c37b74605c2dd0d1090eae47d6707a71954
   pulumi:pulumi:Stack cerulean-cloud-test running Creating installation package.zip ...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip
   pulumi:pulumi:Stack cerulean-cloud-test  9 warnings; 6 messages
Diagnostics:
 pulumi:pulumi:Stack (cerulean-cloud-test):
   Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   Checking Docker is available...
   Building container image...
   Sucessfully built container image with id sha256:285566eca62f43a437d6ad9970241c37b74605c2dd0d1090eae47d6707a71954
   Creating installation package.zip ...
   Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip

   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.

Resources:
   ~ 8 to update
   +-3 to replace
   11 changes. 56 unchanged

… commenting from last commit

jonaraphael

Looks fabulous. All comments universally resolved.

sstill88 added 12 commits January 2, 2025 13:13

added structured_log util and updated github actions file

027da3f

updated structured logs in orchestrator handler

dae4832

Merge branch 'main' into feat/logging-overhaul

f445f5e

towards adding logs to clients.py

840d4c2

add scene_id to model initiation for logging

833f002

Merge branch 'main' into feat/logging-overhaul

a486d20

more logs

7d16538

few more logs

850d769

typo

271151a

maxscale=1

f845076

added notebook and utils to filter logs

656b459

maxScale back to 45

a39c2e6

sstill88 temporarily deployed to test January 9, 2025 00:19 — with GitHub Actions Inactive

sstill88 added 3 commits January 8, 2025 18:39

testing

41c14f5

attempt to fix database schema...

ffe0c4a

attempt to fix database schema take 2...

606d6c9

sstill88 temporarily deployed to test January 9, 2025 02:25 — with GitHub Actions Inactive

sstill88 added 2 commits January 8, 2025 20:38

add sigterm handler with logs

99fe3df

flush logs

d442060

sstill88 temporarily deployed to test January 9, 2025 03:56 — with GitHub Actions Inactive

sstill88 added 2 commits January 8, 2025 21:32

merge/conflict resolution

3c94513

make scene_id global

021bd85

sstill88 temporarily deployed to test January 9, 2025 04:40 — with GitHub Actions Inactive

debugging...

fd329ee

sstill88 temporarily deployed to test January 9, 2025 05:15 — with GitHub Actions Inactive

removed life cycle event logs

3a52dd0

sstill88 temporarily deployed to test January 9, 2025 06:00 — with GitHub Actions Inactive

cleaned up logging in send_inference_request_and_handle_response retries

7d274cf

Shoot. Missed two more.

3ba7fa9

jonaraphael temporarily deployed to test January 26, 2025 20:50 — with GitHub Actions Inactive

jonaraphael reviewed Jan 26, 2025

View reviewed changes

Comment thread notebooks/CloudRunLogs.ipynb Outdated

Comment thread notebooks/CloudRunLogs.ipynb Outdated

try consistent naming

c14a5a6

jonaraphael temporarily deployed to test January 26, 2025 22:17 — with GitHub Actions Inactive

jonaraphael temporarily deployed to test January 27, 2025 05:18 — with GitHub Actions Inactive

jonaraphael reviewed Jan 27, 2025

View reviewed changes

Comment thread cerulean_cloud/cloud_run_orchestrator/handler.py

jonaraphael reviewed Jan 27, 2025

View reviewed changes

Comment thread cerulean_cloud/cloud_run_orchestrator/handler.py

jonaraphael added 6 commits January 27, 2025 23:28

Linting

ee1e0fc

Automatically choose last revision if none provided.

65e433d

Rename start_time

12f28b0

Combine logging utils into one file

3028925

Rename utils.py to structured_logger.py

05507ba

Try fixing the google-cloud-logging import issue in the tests

2e5fa4f

jonaraphael had a problem deploying to test January 28, 2025 05:47 — with GitHub Actions Failure

jonaraphael mentioned this pull request Jan 28, 2025

"You Can't Associate SIGTERM with a Specific Request #138

Open

jonaraphael had a problem deploying to test January 28, 2025 05:52 — with GitHub Actions Failure

Oops. The auto-renaming tool missed this one.

6a703a0

jonaraphael had a problem deploying to test January 28, 2025 05:56 — with GitHub Actions Failure

jonaraphael had a problem deploying to test January 28, 2025 06:05 — with GitHub Actions Failure

test removing logger import

ece53aa

sstill88 temporarily deployed to test January 28, 2025 20:18 — with GitHub Actions Inactive

add google-cloud-logging cloud_run_offset_tiles requirements and undo…

bc7c24f

… commenting from last commit

sstill88 had a problem deploying to test January 28, 2025 20:45 — with GitHub Actions Failure

also add logging to the cloud_run_orchestrator requirements (eye roll)

8d9538b

sstill88 temporarily deployed to test January 28, 2025 21:03 — with GitHub Actions Inactive

updated utils --> structured_logger in CLoudRunLogs.ipynb

48d76ca

jonaraphael approved these changes Jan 28, 2025

View reviewed changes

jonaraphael merged commit 385517c into main Jan 28, 2025

jonaraphael deleted the feat/logging-overhaul branch January 28, 2025 22:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Logging overhaul#137

Logging overhaul#137
jonaraphael merged 47 commits into
mainfrom
feat/logging-overhaul

sstill88 commented Jan 8, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 8, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonaraphael left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sstill88 commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Reviewers

Uh oh!

github-actions Bot commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🍹 preview on cerulean-cloud-images/test

Uh oh!

github-actions Bot commented Jan 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🍹 preview on cerulean-cloud/test

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonaraphael left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sstill88 commented Jan 8, 2025 •

edited

Loading

github-actions Bot commented Jan 8, 2025 •

edited

Loading

🍹 `preview` on cerulean-cloud-images/test

github-actions Bot commented Jan 8, 2025 •

edited

Loading

🍹 `preview` on cerulean-cloud/test