Skip to content

Fix testing perf telemetry: send measurements via measures arg#26028

Open
eleanorjboyd wants to merge 2 commits into
microsoft:mainfrom
eleanorjboyd:fix-testing-perf-telemetry-measures
Open

Fix testing perf telemetry: send measurements via measures arg#26028
eleanorjboyd wants to merge 2 commits into
microsoft:mainfrom
eleanorjboyd:fix-testing-perf-telemetry-measures

Conversation

@eleanorjboyd

Copy link
Copy Markdown
Member

Problem

The discovery/run testing telemetry added in #25980 emitted its numeric performance fields — totalDurationMs, testCount (UNITTEST.DISCOVERY.DONE) and durationMs, requestedCount (UNITTEST.RUN.DONE) — in the properties bag instead of the measures argument of sendTelemetryEvent.

These fields are annotated isMeasurement: true, and telemetry ingestion drops measurement-annotated fields that arrive as properties. Verified against production telemetry: across ~46M unittest.discovery.done events in the last 14 days, zero carried totalDurationMs or testCount (in either Properties or Measures), while the string categoricals (mode, trigger, failureCategory) came through fine. The mode/trigger fields survive because they're strings; the numeric ones silently vanish.

This means there is currently no duration signal to evaluate the recent pytest discovery performance work (#25974, #25982) — the measurements were never actually recorded.

The correct pattern is confirmed by NATIVE_FINDER_PERF, which passes its numbers via the measures (2nd) argument and lands them cleanly in the Measures column.

Fix

Move the numeric measurement fields to the measures (2nd) argument of sendTelemetryEvent at all five emit sites:

  • resultResolver.tsDISCOVERY.DONE success path (totalDurationMs, testCount)
  • controller.tsDISCOVERY.DONE project-error path (totalDurationMs) and RUN.DONE legacy path (durationMs, requestedCount)
  • workspaceTestAdapter.tsDISCOVERY.DONE legacy-error path (totalDurationMs)
  • projectTestExecution.tsRUN.DONE project path (durationMs, requestedCount)

Also removed these fields from the event property type definitions (the __GDPR__ annotations are unchanged and already correctly mark them isMeasurement: true) and added notes so they aren't reintroduced as properties.

Verification

  • tsc: no new type errors from these changes.
  • eslint + prettier: clean on all changed files.
  • Unit tests pass: resolveDiscovery (5), project RUN.DONE telemetry (2), and workspace discovery (5).

Impact

Once a build with this fix ships, totalDurationMs/testCount/durationMs/requestedCount will populate in the Measures column, enabling the before/after discovery-performance comparison (sliced by testCount bucket × mode). No user-facing behavior change.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

The discovery/run telemetry emitted the numeric performance fields
(totalDurationMs, testCount, durationMs, requestedCount) in the
properties bag instead of the measures argument of sendTelemetryEvent.
These fields are annotated `isMeasurement: true`, and telemetry
ingestion drops measurement-annotated fields that arrive as properties.

As a result, across ~46M unittest.discovery.done events none carried a
duration or test-count value, while the string categoricals (mode,
trigger, failureCategory) came through fine. This meant there was no
duration signal to evaluate the recent pytest discovery performance work
(microsoft#25974, microsoft#25982).

Move the numeric fields to the measures (2nd) argument at all five emit
sites so they land in the Measures column, matching the established
NATIVE_FINDER_PERF pattern. Also drop these fields from the event
property type definitions (leaving the __GDPR__ annotations intact) and
add notes so they are not reintroduced as properties.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eleanorjboyd eleanorjboyd self-assigned this Jul 2, 2026
@eleanorjboyd eleanorjboyd added the debt Code quality issues label Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

debt Code quality issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants