W4: source-mapping provenance — grader + census analysis (Gate 1)

.github/workflows/devtools-provenance.yml

@@ -0,0 +1,71 @@
+name: DevTools Provenance (Gate 1)
+
+# Source-mapping (provenance / W4) analysis layer: pure net10.0, no desktop required, so it runs on a
+# hosted runner. Builds the library + analyzer, runs the grader/aggregator/Gate-1 unit tests and the
+# standing Gate-1 check over the committed reference corpus, re-derives the published census rates, and
+# enforces the public-repo scrub. Fresh live-app collection (packaged/trimmed, real-app corpus) is a
+# separate heavy gate that needs an interactive desktop and is not run here.
+
+on:
+  push:
+    branches: [ "winui-devex" ]
+    paths:
+      - "src/winapp-devtools/**"
+      - ".github/workflows/devtools-provenance.yml"
+  pull_request:
+    branches: [ "winui-devex", "main" ]
+    paths:
+      - "src/winapp-devtools/**"
+      - ".github/workflows/devtools-provenance.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  provenance:
+    runs-on: windows-latest
+    defaults:
+      run:
+        shell: pwsh
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v5
+
+      - name: Install .NET
+        uses: actions/setup-dotnet@v5
+        with:
+          dotnet-version: 10.0.x
+
+      # Public-repo hard gate: fail fast on any forbidden internal token before doing anything else.
+      - name: Scrub scan (public-repo forbidden tokens)
+        run: ./src/winapp-devtools/scripts/scrub-scan.ps1
+
+      - name: Build (Release, warnings-as-errors)
+        run: dotnet build src/winapp-devtools/devtools.slnx -c Release
+
+      # Unit tests + the standing Gate-1 check (0% false-confident, Release floors) over the reference corpus.
+      - name: Unit tests + Gate-1 standing check
+        run: >
+          dotnet run --project src/winapp-devtools/provenance/WinApp.DevTools.Provenance.Tests/WinApp.DevTools.Provenance.Tests.csproj
+          -c Release --
+          --report-trx --report-trx-filename provenance.trx
+          --results-directory artifacts/test-results
+
+      # Re-derive the published census rates from the committed corpus and gate on the verdict
+      # (--strict makes CONDITIONAL/INCONCLUSIVE non-zero too; a KILL is always non-zero).
+      - name: Census analyze + Gate-1 verdict
+        run: >
+          dotnet run --project src/winapp-devtools/provenance/WinApp.DevTools.Provenance.Census/WinApp.DevTools.Provenance.Census.csproj
+          -c Release --
+          analyze src/winapp-devtools/provenance/WinApp.DevTools.Provenance.Tests/Fixtures/census
+          --out artifacts/census --strict
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: provenance-census
+          path: |
+            artifacts/test-results/**
+            artifacts/census/**

specs/winapp-devtools-provenance.md

@@ -1,6 +1,10 @@
 # Spec: provenance — confidence-graded source mapping & the census (W4)
 
 > **Status:** 🟡 Draft v0.1 — best-effort, **honestly graded**, and gated by a pre-build reality census.
+> **Implementation:** the pure-logic **analysis layer landed** — grader, census analysis + Gate-1
+> evaluator, unit tests, and the standing Gate-1 check ship in `src/winapp-devtools/` (see §8). Live
+> collection + the live `Source.resolve`/`census` commands are deferred (heavy gate — need the running-app
+> daemon and the W3 read floor).
 > **Branch:** `winui-devex` · **Owner:** (you) · **Workstream:** W4
 > **Related:** `winapp-devtools-protocol.md` (the `Source` family + `SourceKind` / `Confidence` enums) ·
 > `winapp-devtools-read.md` (the guaranteed floor this annotates) · `winapp-devtools-hot-reload.md`
@@ -147,6 +151,26 @@ prohibited; the census is a pre-build funding gate; persist (W5) is confidence-g
    anything that *depends* on provenance.
 4. **Persist integration.** Expose the confidence W5 needs; enforce the persist threshold.
 
+### Implementation status
+
+The **analysis layer** for phases 1–3 has landed as pure `net10.0` (CI-runnable, no desktop) under
+`src/winapp-devtools/`:
+
+- **Grade (phase 1).** `SourceProvenanceGrader` implements the §4 honesty model; fixture unit tests
+  cover every §4 case including the two **false-confident-prohibition** cases. The grader never emits
+  `exact` unless the origin is authored markup **with** a resolved line.
+- **Census analysis + Gate 1 (phases 2–3, analysis half).** `CensusTsvReader` → `CensusAggregator`
+  (counts by `SourceKind`) → `Gate1Evaluator` (§5 kill-criteria) → `CensusReport` (md + json). A
+  console analyzer (`analyze <tsvDir>`) and a scrubbed orchestration harness (`census/Run-Census.ps1`)
+  drive it. Published rates from the committed **reference corpus** live in `census/results/`; a
+  standing Gate-1 MSTest check + the `devtools-provenance.yml` workflow enforce **0 false-confident**
+  on every run.
+
+**Deferred (heavy gate / needs other workstreams):** live `Source.resolve`/`census` commands wired
+into the running-app daemon; **fresh** collection (injecting the read component; packaged/trimmed
+configs; a real-app corpus per **Q-CENSUS-CORPUS**) — needs an interactive desktop; W5 persist
+integration (phase 4).
+
 ## Appendix — where W4 sits
 
 ```mermaid

src/winapp-devtools/.gitignore

@@ -0,0 +1,2 @@
+# CI / local scratch output (published rates live in census/results, which IS committed)
+artifacts/

src/winapp-devtools/Directory.Build.props

@@ -0,0 +1,27 @@
+<Project>
+
+  <!--
+    Shared build settings for the winapp DevTools workstreams.
+    These projects are intentionally pure net10.0 (no -windows / no WinUI) so the
+    provenance grading + census-analysis logic and its unit tests run on hosted CI
+    without an interactive desktop. The desktop-only census *collection* harness lives
+    under census/ and is a documented heavy gate.
+  -->
+
+  <PropertyGroup>
+    <TargetFramework>net10.0</TargetFramework>
+    <Nullable>enable</Nullable>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <LangVersion>latest</LangVersion>
+    <EnableNETAnalyzers>true</EnableNETAnalyzers>
+    <AnalysisMode>Recommended</AnalysisMode>
+    <Copyright>Copyright (C) Microsoft Corporation and Contributors</Copyright>
+    <Company>Microsoft Corporation</Company>
+    <NeutralLanguage>en-US</NeutralLanguage>
+  </PropertyGroup>
+
+  <PropertyGroup Condition="'$(Configuration)' == 'Release'">
+    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
+  </PropertyGroup>
+
+</Project>

src/winapp-devtools/README.md

@@ -0,0 +1,24 @@
+# winapp devtools
+
+Home for the **DevTools** workstreams — the CLI-driven capabilities for inspecting and editing running
+Windows app UIs. Each workstream has a spec under `specs/winapp-devtools-*.md`.
+
+## What's here
+
+| Area | Workstream | Status |
+|---|---|---|
+| `provenance/` + `census/` | **W4 — source mapping / provenance** ("which line of source produced this live element?") | analysis layer landed (pure `net10.0`, CI-runnable) |
+
+See `provenance/` for the confidence-graded source-mapping logic and `census/README.md` for the Gate-1
+resolution census. Everything here is pure `net10.0` (no WinUI / no `-windows` TFM) so it builds and
+tests on a hosted runner without a desktop; the live-app collection half is a documented heavy gate.
+
+## Build & test
+
+```powershell
+dotnet build src/winapp-devtools/devtools.slnx -c Release
+dotnet test  src/winapp-devtools/devtools.slnx
+```
+
+The `devtools-provenance.yml` workflow builds Release, runs the unit tests + the standing Gate-1 check,
+re-derives the published census rates, and enforces the public-repo scrub (`scripts/scrub-scan.ps1`).

src/winapp-devtools/census/README.md

@@ -0,0 +1,70 @@
+# Source-resolution census (Gate 1)
+
+The **census** measures how often source mapping *actually* resolves a live element to its authored
+source across build configs — the cheapest way to falsify the provenance plan **before** anything is
+funded on top of it. See `specs/winapp-devtools-provenance.md` (§5) for the kill-criteria.
+
+## Two halves: collect vs. analyze
+
+The census is deliberately split so the measurement half runs anywhere:
+
+| Half | What it does | Where it runs |
+|---|---|---|
+| **Collect** | Launch a target app in a given config, enumerate its live element tree, and write one TSV row per element (`handle, type, name, file, line, col`). | **Heavy gate** — needs an interactive desktop and the running-app read component (owned by the "reading the UI" workstream; not in this repo yet). |
+| **Analyze** | Grade every row (`SourceKind` + `Confidence` + `ReasonCode`), aggregate by `SourceKind`, apply the Gate-1 kill-criteria, and emit the report + verdict. | **Pure `net10.0`** — hosted CI, no desktop. This is the single source of truth for grading and the verdict. |
+
+Only **analyze** is exercised on CI. It runs over a committed **reference corpus** of scrub-clean TSVs
+collected from the proof-of-concept, so we publish real rates without a desktop.
+
+## Run the analysis (no desktop required)
+
+```powershell
+# Grade + aggregate + Gate-1 verdict over a directory of census TSVs, writing md + json:
+dotnet run --project ../provenance/WinApp.DevTools.Provenance.Census/WinApp.DevTools.Provenance.Census.csproj -- `
+  analyze ../provenance/WinApp.DevTools.Provenance.Tests/Fixtures/census --out results
+
+# Exit codes: 0 = GO, 2 = KILL. With --strict, CONDITIONAL/INCONCLUSIVE are non-zero (3) too.
+```
+
+`results/census-latest.md` / `.json` are the committed baseline (regenerated by CI). The
+orchestration harness `Run-Census.ps1 -AnalyzeOnly` wraps the same call.
+
+## Collect fresh data (heavy gate)
+
+`Run-Census.ps1` also carries the config matrix (Debug / Release / packaged / trimmed) and a **gated
+`-Collector` seam**. Fresh collection is a documented heavy-gate step: it needs an interactive desktop
+and the read component to enumerate a running app. Point the harness at that collector when it lands:
+
+```powershell
+./Run-Census.ps1 -Collector <path-to-read-component-collector> -Configs Debug,Release
+```
+
+Until then, collection is out of scope for CI and the reference corpus stands in for the local run.
+
+## Gate-1 kill-criteria (spec §5)
+
+| Metric | Floor / rule |
+|---|---|
+| source-backed elements resolved in **Release** | **≥ 70%** |
+| templated/generated resolved to template in Release | **≥ 40%** |
+| **false-confident** (`exact`/`high` that was wrong) | **0% — any hit is an automatic KILL** |
+
+The false-confident audit is computed **independently of the grader** (it re-checks each confident row
+against whether the file is authored markup), so a grader regression that starts over-trusting
+framework/template/runtime rows is caught rather than hidden. This runs as a standing MSTest check and
+in the `devtools-provenance.yml` workflow on every change.
+
+## Reference corpus baseline
+
+From the committed corpus, the non-stripping Release-family config resolves **100%** of source-backed
+elements, maps **58.3%** of templated/generated elements to their template/style, with **0**
+false-confident → **GO** (Debug identical; the line-info-stripped config drops authored+line to 0% as
+expected, which is why it is diagnostic-only and never the Gate-1 arbiter).
+
+## Open question — Q-CENSUS-CORPUS
+
+The committed TSVs are a **single-app** reference corpus — enough to exercise the analysis and lock the
+baseline, **not** a representative measurement of the ecosystem. What real apps (number + mix:
+first-party, community, templated-heavy, `x:Bind`-heavy, packaged, trimmed) should constitute the
+standing corpus is **open** and tracked as `Q-CENSUS-CORPUS` in the spec. Producing those numbers is
+part of the heavy-gate collection follow-up.

src/winapp-devtools/census/Run-Census.ps1

@@ -0,0 +1,124 @@
+<#
+.SYNOPSIS
+  Source-resolution census (Gate 1) harness. Orchestrates the census across a build-config matrix and
+  hands the collected per-element data to the tested analysis layer, which grades every element,
+  aggregates by SourceKind, and returns a GO / CONDITIONAL / KILL / INCONCLUSIVE verdict.
+
+.DESCRIPTION
+  Productization deliberately splits "collect" from "analyze":
+
+    * ANALYZE (this harness drives it; pure, CI-runnable) — the winapp-provenance-census tool reads a
+      directory of census TSVs (handle, type, name, file, line, col), applies the source-provenance
+      grader (spec section 4) and the Gate-1 kill-criteria (spec section 5), and writes
+      results/census-latest.md + results/census-latest.json.
+
+    * COLLECT (heavy gate; NOT in this repo) — producing a fresh TSV means launching a real app and
+      reading its live UI element tree with the required runtime component. That collector is owned by
+      the "reading the UI" workstream, needs an interactive desktop, and is supplied to this harness
+      through the -Collector seam. Until it lands, run the harness in -AnalyzeOnly mode over an
+      existing corpus of TSVs (e.g. the committed reference corpus under results/).
+
+  This keeps the honesty model and the verdict in one tested place (the analyzer) instead of a script.
+
+.PARAMETER Pages
+  Fixture page labels that make up one census sweep (used only when collecting).
+
+.PARAMETER Configs
+  Build-config labels to sweep. The analyzer judges Gate-1 on the non-stripping Release config;
+  'release-nolineinfo' is the diagnostic line-info-stripped probe (never the arbiter). 'packaged' and
+  'trimmed' are defined for the full matrix but need desktop/packaged collection.
+
+.PARAMETER TsvDir
+  Directory the census TSVs are read from (defaults to ResultsDir).
+
+.PARAMETER ResultsDir
+  Directory the published report (census-latest.md/json) is written to.
+
+.PARAMETER Collector
+  Heavy-gate seam: a script block invoked as & $Collector $configLabel $page $outTsv to collect one
+  TSV from a running app via the required runtime component. Omit to run analyze-only.
+
+.PARAMETER FixtureProject
+  Optional path to an app project to build per config before collecting (owned by the reading-the-UI
+  workstream; not required for analyze-only runs).
+
+.PARAMETER AnalyzeOnly
+  Skip collection entirely and (re)analyze the TSVs already in TsvDir. This is the CI / no-desktop path.
+
+.EXAMPLE
+  # No-desktop: re-analyze the committed reference corpus and refresh the published rates.
+  pwsh src/winapp-devtools/census/Run-Census.ps1 -AnalyzeOnly `
+      -TsvDir src/winapp-devtools/provenance/WinApp.DevTools.Provenance.Tests/Fixtures/census
+
+.EXAMPLE
+  # Heavy gate (desktop): collect fresh TSVs with an operator-supplied collector, then analyze.
+  pwsh src/winapp-devtools/census/Run-Census.ps1 -Collector $myCollector -FixtureProject path/to/app
+#>
+[CmdletBinding()]
+param(
+    [string[]]$Pages   = @('SmokePage', 'Repeater', 'Items', 'UcHost', 'XBindFn'),
+    [string[]]$Configs = @('debug', 'release', 'release-nolineinfo'),
+    [string]$TsvDir,
+    [string]$ResultsDir,
+    [scriptblock]$Collector,
+    [string]$FixtureProject,
+    [switch]$AnalyzeOnly
+)
+
+$ErrorActionPreference = 'Stop'
+$root = $PSScriptRoot
+$ResultsDir = if ($ResultsDir) { $ResultsDir } else { Join-Path $root 'results' }
+$TsvDir = if ($TsvDir) { $TsvDir } else { $ResultsDir }
+$analyzerProj = Join-Path $root '..\provenance\WinApp.DevTools.Provenance.Census\WinApp.DevTools.Provenance.Census.csproj'
+
+New-Item -ItemType Directory -Force $ResultsDir | Out-Null
+
+# Build-config matrix. release-nolineinfo forces XAML line-info off (DisableXbfLineInfo is a public,
+# supported switch) so we can measure the worst-case "line-info stripped" behaviour.
+$configMap = @{
+    'debug'              = @{ Configuration = 'Debug';   Props = @() }
+    'release'            = @{ Configuration = 'Release'; Props = @() }
+    'release-nolineinfo' = @{ Configuration = 'Release'; Props = @('-p:DisableXbfLineInfo=true') }
+    'packaged'           = @{ Configuration = 'Release'; Props = @() }
+    'trimmed'            = @{ Configuration = 'Release'; Props = @('-p:PublishTrimmed=true') }
+}
+
+function Build-Fixture([string]$label) {
+    if (-not $FixtureProject) { return }
+    $c = $configMap[$label]
+    if (-not $c) { throw "unknown config label '$label'" }
+    Write-Host "==> building fixture [$label] ($($c.Configuration)) $($c.Props -join ' ')" -ForegroundColor Cyan
+    $buildArgs = @('build', $FixtureProject, '-c', $c.Configuration, '--nologo') + $c.Props
+    & dotnet @buildArgs 2>&1 | Select-Object -Last 3 | ForEach-Object { Write-Host "    $_" -ForegroundColor DarkGray }
+    if ($LASTEXITCODE -ne 0) { throw "fixture build failed for $label" }
+}
+
+# ---------------- collect (heavy gate; only when a collector is supplied) ----------------
+if (-not $AnalyzeOnly -and $Collector) {
+    foreach ($label in $Configs) {
+        Build-Fixture $label
+        foreach ($page in $Pages) {
+            $outTsv = Join-Path $TsvDir "$label-$page.tsv"
+            Write-Host "==> collecting [$label/$page] -> $(Split-Path $outTsv -Leaf)" -ForegroundColor Cyan
+            & $Collector $label $page $outTsv
+            if ($LASTEXITCODE -ne 0) { Write-Host "    collector failed for $label/$page" -ForegroundColor Red }
+        }
+    }
+}
+elseif (-not $AnalyzeOnly) {
+    Write-Host "==> no -Collector supplied; running analyze-only over existing TSVs in $TsvDir" -ForegroundColor Yellow
+    Write-Host "    (fresh collection is a heavy gate: it needs a desktop + the reading-the-UI collector)" -ForegroundColor DarkGray
+}
+
+# ---------------- analyze (pure; delegates grading + verdict to the tested library) ----------------
+if (-not (Get-ChildItem -Path $TsvDir -Filter *.tsv -ErrorAction SilentlyContinue)) {
+    throw "no census TSVs found in $TsvDir - collect some (heavy gate) or point -TsvDir at a corpus (e.g. the reference corpus)."
+}
+
+Write-Host "==> analyzing census TSVs in $TsvDir" -ForegroundColor Cyan
+& dotnet run --project $analyzerProj -c Release -- analyze $TsvDir --out $ResultsDir
+$verdictExit = $LASTEXITCODE
+
+Write-Host ""
+Write-Host "report: $(Join-Path $ResultsDir 'census-latest.md')" -ForegroundColor Cyan
+exit $verdictExit