Skip to content

fix(cli): stop telemetry hangs and errors on blocked networks#5880

Open
pamelachia wants to merge 8 commits into
developfrom
pamela/cli-telemetry-network-resilience
Open

fix(cli): stop telemetry hangs and errors on blocked networks#5880
pamelachia wants to merge 8 commits into
developfrom
pamela/cli-telemetry-network-resilience

Conversation

@pamelachia

@pamelachia pamelachia commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

On networks that block or blackhole eu.posthog.com (corporate proxies, CI and Docker egress policies, air-gapped machines), CLI telemetry currently breaks the CLI itself. Reproduced on stable v2.109.1 with the PostHog host pointed at an unreachable address: every command pays a 5s trailing hang and then exits 1 with {"code":"UnknownError","message":"Timeout while shutting down PostHog. Some events may not have been sent."}, even though the command's real work succeeded. Long-running commands additionally print raw SDK stack traces to stderr, and the Go binary's shutdown wait is unbounded. The only workaround is supabase telemetry disable, which permanently removes those users from CLI analytics. This PR makes telemetry strictly fire-and-forget: a failed delivery never changes a command's output, exit code, or runtime beyond a ~2s cap.

Changes

  • Go binary: set a 2s ShutdownTimeout on the PostHog client (the SDK default waits indefinitely for the final flush, including retries) and route SDK logs to the --debug logger instead of unconditional stderr.
  • TS shells: new shared scopedPosthogClient factory in shared/telemetry/posthog-client.ts, used by both analytics.layer.ts and legacy-analytics.layer.ts. It wraps fetch to report delivery failures as delivered: posthog-node has no logger hook and prints failures through hardcoded console.error calls, so making the transport infallible from the SDK's perspective is the one supported way to get no retries, no stderr output, and no shutdown stall. A 2s requestTimeout bounds how long a blackholed connection can hold exit.
  • Root cause of the exit-1 failure: the shutdown finalizer was Effect.promise(() => client._shutdown(5_000)).pipe(Effect.ignore), but Effect.promise turns a promise rejection into a defect and Effect.ignore only swallows typed failures, so a shutdown timeout failed the whole command. The rejection is now caught on the promise itself, so telemetry shutdown can never fail a command.

Context for review

  • flushAt: 1, flushInterval: 0 is kept deliberately. For a short-lived CLI process, eager flush overlaps the send with command execution; batching would move the network wait to process exit where the user feels it.
  • The exit bounds are caps, not floors: on healthy networks both shells return as soon as the flush completes. Fully detaching the send instead would race process exit and silently drop events on healthy networks, which is the common case.
  • Verified against real unreachable endpoints. Refused connection: stable v2.109.1 runs telemetry status in 5s with exit 1 and the UnknownError payload; this branch runs it instantly with exit 0 and empty stderr. Blackholed (non-routable address): Go Close() returns in 2.0s (previously unbounded), TS capture plus shutdown completes in 2.0s with zero stderr bytes.
  • Accepted tradeoff: on working networks slower than 2s round-trip to PostHog, events drop. Telemetry is best-effort.
  • Delivery failures remain observable under --debug on the Go side. On the TS side they are dropped silently.

Linear

  • fixes GROWTH-1001

@pamelachia
pamelachia requested a review from a team as a code owner July 15, 2026 06:03
@pamelachia pamelachia self-assigned this Jul 15, 2026
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@8bec921a428a067c1621fd5b4b54474f80841eb2

Preview package for commit 8bec921.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a96c6f5826

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e38cb95741

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli-go/internal/telemetry/client.go
@avallete

Copy link
Copy Markdown
Member

@codex review

@pamelachia
pamelachia marked this pull request as draft July 15, 2026 11:21
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 77ce8398ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pamelachia
pamelachia marked this pull request as ready for review July 17, 2026 06:04
Comment on lines +21 to +35
export const scopedPosthogClient = (apiKey: string, host: string) =>
Effect.acquireRelease(
Effect.sync(
() =>
new PostHog(apiKey, {
host,
flushAt: 1,
flushInterval: 0,
requestTimeout: 2_000,
fetch: fireAndForgetFetch,
}),
),
// Catch on the promise: Effect.promise turns rejections into defects,
// which escape Effect.ignore and fail the command.
(client) => Effect.promise(() => client._shutdown(5_000).catch(() => undefined)),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue one event can be in flight while a later exit event remains queued. Shutdown then performs two sequential 2-second requests (eg: shutdownMs: 3987 with two captures against the pinned SDK). Apply the deadline to the entire shutdown and add a two-event blackhole test.

The existing port-9 test rejects immediately and misses this case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants