out_stackdriver: service account authentication broken in v5.0.0

[zpoint]

Bug Report

Describe the bug

The stackdriver output plugin's service account authentication is completely broken in v5.0.0. Both google_service_credentials config option and GOOGLE_APPLICATION_CREDENTIALS env var are ignored — the plugin only tries the GCE metadata server, which fails on non-GCP environments (AWS, Azure, on-prem, etc.).

This is a regression from v4.2.3, where the same config works correctly.

To Reproduce

Run fluent-bit v5.0.0 on any non-GCE machine with a valid GCP service account JSON key:

[INPUT]
    name dummy

[OUTPUT]
    name stackdriver
    match *
    google_service_credentials /path/to/service-account-key.json

Or equivalently in YAML format:

pipeline:
  inputs:
  - name: dummy
  outputs:
  - name: stackdriver
    match: "*"
    google_service_credentials: /path/to/service-account-key.json

Expected behavior

fluent-bit reads the service account key from the specified file and authenticates via OAuth2 JWT, as documented in https://docs.fluentbit.io/manual/data-pipeline/outputs/stackdriver

Actual behavior

fluent-bit ignores the credentials file and only tries the GCE metadata server:

[output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[error] [output:stackdriver:stackdriver.0] error retrieving oauth2 access token
[error] [output:stackdriver:stackdriver.0] cannot retrieve oauth2 token
[warn] [engine] failed to flush chunk '...' cannot be retried

All log chunks are dropped after retry exhaustion.

Comparison with v4.2.3

Same config, same credentials file, same machine:

• v5.0.0: error retrieving oauth2 access token ❌
• v4.2.3: [oauth2] access token from 'oauth2.googleapis.com:443' retrieved ✅

Environment

• fluent-bit version: 5.0.0 (from packages.fluentbit.io apt repo, Ubuntu 24.04)
• OS: Ubuntu 24.04 (AWS EC2)
• Credential type: GCP service account key JSON ("type": "service_account")

Possible cause

The OAuth2 core library (flb_oauth2.h) was significantly refactored between v4.2.3 and v5.0.0 (new flb_oauth2_config struct, JWT support for OpenTelemetry). The stackdriver plugin code itself was not changed, suggesting the regression is in the shared OAuth2 layer's handling of service account JWT auth.

Workaround

Pin fluent-bit to v4.x: sudo apt-get install -y "fluent-bit=4.*"

[zpoint]

For reference, this broke our CI at skypilot-org/skypilot#9185 — we use the stackdriver plugin to ship logs from AWS/Azure clusters to GCP Cloud Logging. We've pinned to v4.x as a workaround.