Add Datadog CLI-usage dashboard + JSON validity gate#67
Merged
Conversation
Drop the `assembly` half-block wordmark from the bare-command welcome screen; `print_banner()` now emits only the one-line version + tagline header. Update the onboard tests to assert on the tagline header instead of the block glyphs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Importable dashboard (docs/datadog/cli-usage-dashboard.json) + README, built on the source:aai-cli logs from aai_cli/telemetry.py: adoption, reliability/ errors, performance, and environment widgets. No template variables so it renders without facet setup; grouped widgets populate once facets exist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/json_lint.py parses every tracked + staged *.json (committed dashboards, init-template vercel.json, recorded API fixtures) so a malformed file fails the gate loudly instead of silently downstream. Validity only -- the recorded tests/fixtures/api/*.json snapshots are not reformatted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related additions, prompted by a long debugging session into "no telemetry data showing in Datadog" for the
aaiCLI (the telemetry shipped in #66).1. Datadog dashboard (
docs/datadog/)cli-usage-dashboard.json— importable dashboard built on thesource:aai-clilogs fromaai_cli/telemetry.py. Four groups: Adoption & Usage (runs, unique devices, top commands, version adoption), Reliability & Errors (error rate, outcomes over time, top error types, exit codes), Performance (duration p50/p90/p95, slowest commands), Environment & Reach (OS, CI vs interactive, Python/CLI version).README.mddocuments import + the facet prerequisite and caveats (error-rate includes expected user errors;@device_id≈ installs not people; ~1–2 min indexing lag).2. JSON validity gate (
scripts/json_lint.py+scripts/check.sh)scripts/json_lint.pyparses every tracked + staged*.json(committed dashboards, init-templatevercel.json, recorded API fixtures) so a malformed file fails the gate loudly instead of silently downstream (a bad dashboard just won't import).check.shjust before the prettier step. Validity only — the recordedtests/fixtures/api/*.jsonsnapshots are not reformatted.Context / what the debugging found
The "no data" symptom was not a code bug: the CLI sender and event shape are correct (verified a probe with the full
@datadog/browser-logsquery-param set behaves identically). Root cause was org-side — the Datadog org had no log index, soaai-clilogs were ingested (visible in Live Tail) but never indexed (invisible to dashboards/search). Resolved by creating asource:aai-cli-scoped index; dashboard now populates.Testing
./scripts/check.sh→ All checks passed. (new JSON gate runs as part of it; validates 15 JSON files).json_lint.pyexits 1 on a malformed JSON file.source:aai-clidata.🤖 Generated with Claude Code