Skip to content

Automate public API diff reports for in-development .NET releases#10469

Open
jeffhandley wants to merge 8 commits into
dotnet:mainfrom
jeffhandley:jeffhandley/api-diff-orchestrator
Open

Automate public API diff reports for in-development .NET releases#10469
jeffhandley wants to merge 8 commits into
dotnet:mainfrom
jeffhandley:jeffhandley/api-diff-orchestrator

Conversation

@jeffhandley

@jeffhandley jeffhandley commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Adds an "API Diff" automation that keeps public API diff report PRs continuously
up to date for the active, not-yet-shipped .NET majors. A scheduled orchestrator
deterministically discovers which diffs should exist from the dnceng NuGet feeds
and dispatches a per-target worker that generates the reports from real published
builds and opens/maintains a draft PR for each one. The automation never
merges -- humans review and merge.

This PR adds only the automation and tooling. No API diff report content is
included; the reports are produced as draft PRs by the workflows after this
merges.

How it works

Discovery (deterministic, no agent). api-diff-orchestrator.yml is a plain
workflow whose discover job runs a C# file-based app, api-diff-discover.cs,
that uses the NuGet client SDK (NuGet.Protocol + NuGet.Versioning) for
SemVer-correct feed queries. maxGA is the highest GA major on dotnet-public;
only maxGA+1 and maxGA+2 are considered. It emits one target per diff, each
pinning the exact previous/current package versions and their feeds, and fans out
to the API Diff Worker workflow via workflow_call (max-parallel 1, matching the
worker's shared concurrency group so a queued target is never dropped).

Two tracks of diffs are produced:

  • Major-to-major (cumulative). Current = newest dotnet{major} dev-feed build
    of any prerelease label; baseline = newest (major-1) RC-or-GA build on
    dotnet-public, advancing RC1 -> RC2 -> GA. Identity is keyed on the major and
    kept canonical ((major-1).0-ga_to_major.0) so one evergreen PR is reused as
    the head advances. Content at release-notes/<major>.0/<major>.0.0/api-diff.
  • Incremental (milestone deltas). Head is sourced from the dev feed so a
    milestone's draft PR is produced as soon as its build exists (to merge
    before the milestone releases); the baseline is pinned to the previous
    milestone's shipped public build, so a milestone's diff holds off until its
    baseline has shipped. Content under
    release-notes/<major>.0/preview/<milestone>/api-diff.

Generation + publishing (agentic worker). The API Diff Worker workflow,
api-diff-worker.md (compiled to api-diff-worker.lock.yml), is the per-target
worker. Its host generation script, api-diff-generate.sh, positions the working
tree on the publish branch and runs the api-diff skill's RunApiDiff.ps1 against
the target's pinned versions/feeds, gating on completeness (a non-zero generation
exit publishes nothing). The read-only agent then commits the generated
reports and publishes entirely through gh-aw native safe outputs:
create-pull-request / push-to-pull-request-branch open or append to the draft
PR, update-pull-request refreshes the title and body, add-comment posts a run
summary, mark-pull-request-as-ready-for-review flips it Ready when the milestone
is code-complete, and noop covers the silent path. A post-step re-validates the
agent-written state block against the host-authored target before anything is
published.

Safety and robustness

  • Draft PRs only; never merges. Humans own review and merge.
  • Least privilege. The agent job is read-only; every GitHub write runs in a
    separate, permission-controlled safe-output job constrained to the automation
    • api-diff labels and an allowed-files allowlist scoped to the api-diff
      report tree plus the two global exclusion files (release-notes/ApiDiff*ToExclude.txt).
      Nothing outside that allowlist can be written; the default protected-file check
      is kept, excluding only the per-folder api-diff/README.md index the reports
      legitimately produce.
  • Fork-safe. Discovery and the worker are gated to the canonical repo; a fork
    runs them only via manual workflow_dispatch, so forks never open spurious PRs.
  • Idempotent. An unchanged build is a silent no-op; identity is anchored to a
    visible YAML state marker in the PR body so runs never create duplicates.
  • Human bootstrapping. A person can hand-start an API diff PR (on the target
    branch, or an opted-in automation + api-diff PR) and the automation adopts
    and maintains it thereafter. A PR labeled only api-diff stays human-owned and
    blocks the automation for that content.
  • Reviewer feedback. The worker folds in reviewer feedback on its next
    scheduled run using GitHub integrity filtering (min-integrity: approved), so
    only write-access or maintainer-endorsed feedback reaches the agent; it never
    triggers on review events and applies no trust handling of its own.

Commit walkthrough

  1. Upgrade gh-aw to v0.80.9 -- prerequisite tooling bump. Renames the
    dispatcher agent, adds its skill, and recompiles existing workflows, so this
    commit also regenerates unrelated lock files (release-notes.lock.yml,
    agentics-maintenance.yml, action pins). Tooling-only churn.
  2. Refactor api-diff skill into pure generation plus validation -- api-diff
    becomes pure report generation (RunApiDiff.ps1); a new api-diff-validation
    skill owns the dotnet-inspect verification. RunApiDiff.ps1 accepts any
    <label>.N prerelease build and hardens temp-folder cleanup.
  3. Add API Diff Orchestrator workflow -- deterministic discovery
    (api-diff-orchestrator.yml + api-diff-discover.cs) and the two-track model.
  4. Add API Diff Worker workflow -- the per-target worker (api-diff-worker.md
    / api-diff-worker.lock.yml) and its host generation script
    (api-diff-generate.sh), publishing through gh-aw native safe outputs.
  5. Adopt human-bootstrapped API diff PRs -- adoption rules for hand-started
    PRs.
  6. Add support for incorporating PR feedback -- integrity-filtered reviewer
    feedback handling and the delimited state block.

Two trailing commits satisfy super-linter on the host shell script: marking
api-diff-generate.sh executable (bash-exec) and an shfmt formatting fix.

Notes for reviewers

  • All agentic jobs run in the repo's copilot-pat-pool environment per the
    existing PAT pool mechanism.
  • Validated end-to-end on a simulation fork of this repository:
    from an empty net11 state it produces the major-to-major draft PR plus the full
    preview.1 through preview.6 incremental draft PRs, correctly holding off
    preview.7 until its baseline ships. The linked pull requests show the generated
    draft PRs and the commit-by-commit progression as each build advanced.

jeffhandley and others added 2 commits July 7, 2026 18:02
- Rename dispatcher agent .github/agents/agentic-workflows.agent.md to
  agentic-workflows.md and re-add the repo-specific PAT pool, schedule
  seed, and frontmatter ordering instructions
- Add dispatcher skill .github/skills/agentic-workflows/SKILL.md
- Recompile release-notes with v0.80.9 and add the generated
  agentics-maintenance.yml
- Apply safe-outputs codemod in release-notes.md: required-title-prefix
  and required-labels for push-to-pull-request-branch
- Bump pinned actions in .github/aw/actions-lock.json: actions/checkout
  v7.0.0, actions/setup-dotnet v5.4.0, github/gh-aw-actions/setup v0.80.9
- Update copilot-setup-steps.yml gh-aw setup-cli to v0.80.9
- Update validate-pat-pool.yml github/gh-aw-actions/setup pin to v0.80.9 (8c7d04e)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Make the api-diff skill pure report generation (RunApiDiff.ps1 only)
- Add api-diff-validation skill owning the dotnet-inspect verification that
  confirms diffed APIs actually shipped (moved out of api-diff)
- Light-touch update release-notes and generate-changes references to use
  api-diff for diffs and api-diff-validation for verification
- Harden RunApiDiff.ps1 temp-folder cleanup with a bounded-retry recursive
  delete so a transient "directory not empty" race on Linux no longer aborts a
  multi-SDK diff run
- Accept any "<label>.N" prerelease version in RunApiDiff.ps1 instead of only
  preview/rc/ga, keeping RC and GA specially cased and treating every other
  label (alpha, beta, ...) as a plain semantic-version label, so a diff can be
  generated against any published build. Updates the api-diff skill mapping and
  RunApiDiff.md accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeffhandley jeffhandley force-pushed the jeffhandley/api-diff-orchestrator branch from 8e64de0 to 2032529 Compare July 8, 2026 00:42
jeffhandley and others added 4 commits July 8, 2026 02:11
Deterministic discovery (no agent) driven entirely by the dnceng NuGet feeds that
decides which public API diffs to produce for the active, not-yet-shipped majors,
emitting one target per diff.

- api-diff-orchestrator.yml: a plain (non-agentic) orchestration workflow whose discover
  job sets up the .NET SDK and runs the discovery app, capturing its JSON from a file
  so the "dotnet run" build output on stdout cannot contaminate the result.
- api-diff-discover.cs: a C# file-based app using the NuGet client SDK (NuGet.Protocol
  + NuGet.Versioning) for SemVer-correct feed version discovery. maxGA is the highest GA
  major on dotnet-public; only maxGA+1 and maxGA+2 are considered.
- major-to-major (cumulative): current build = newest dotnet{major} build of any
  prerelease label; baseline = newest (major-1) build on dotnet-public that is RC-or-GA,
  advancing RC1 -> RC2 -> GA. Identity is keyed on the major only and kept canonical
  ((major-1).0-ga_to_major.0) so one evergreen PR is reused as the head advances. Content
  at the major root release-notes/major.0/major.0.0/api-diff.
- incremental (milestone deltas): previous public milestone -> latest public milestone,
  both named public milestones (preview.N / rc.N) on dotnet-public. Content under
  release-notes/major.0/preview/<milestone>/api-diff.
- Each target pins the exact previous/current package versions and their feeds.
- Status: maxGA+1 is code-complete once maxGA+2 has started publishing builds; otherwise
  in-development. maxGA+2 is in-development.
- Retry feed queries so a transient blip is not read as "no build"; the maxGA+2 channel
  probe is quiet since it does not exist until main forks to it.
- Fix RunApiDiff.ps1 string-sorting bugs where builds with 3-digit identifiers
  (e.g., preview.1.100.1) sorted incorrectly relative to 2-digit ones. The build
  suffixes are zero-padded during sort to ensure proper semantic ordering when
  picking the latest build from feeds.
  (Note: It is OK that we only see the first 100 files via `gh pr list --json files`
  as api-diff PRs are focused to the files we're looking for.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per-target worker (invoked by the orchestrator via workflow_call, or manually) that
generates a milestone's public API diff from real published builds and
opens/maintains its draft pull request. It never merges; humans merge.

- api-diff-orchestrator.yml: add the api-diff job that fans out over the discovered targets
  and calls the worker as a reusable workflow (workflow_call) so it runs in the
  orchestrator's run context and inherits the orchestrator's actor, satisfying the worker's
  gh-aw activation role check. Runs the matrix with max-parallel 1, matching the
  worker's shared concurrency group so a queued target is never dropped by
  concurrency's pending-run-replacement rule.
- api-diff-generate.sh: host-side setup runs the api-diff skill's
  RunApiDiff.ps1 against the target's pinned versions/feeds (current build from
  the dotnet{major} channel feed, baseline from dotnet-public; the ApiDiff tool
  from dotnet{major}-transport). Relocates major-to-major output into <mm>/<mm>.0
  and gates on completeness: a non-zero RunApiDiff exit publishes nothing, and a
  build that produced at least one SDK report (NETCore / AspNetCore /
  WindowsDesktop) publishes those, since RunApiDiff fails outright when an SDK's
  ref packs cannot be downloaded. Emits a per-run step summary covering every
  outcome, including no-op.
- Identity + guards: find this diff's PR by matching a visible YAML marker as a
  whole comment line in the body (so a cumulative marker cannot hijack an
  incremental first-preview PR whose marker contains it as a prefix); resolve
  candidates from the api-diff label unioned with any PR already on the target
  branch (or collision branches starting with the target branch via search), so a
  label a human stripped never causes a duplicate. Stand down when a non-automation
  api-diff PR already owns the content_dir (verified via `gh pr view --json files`
  to avoid API truncation limits).
- No-op / metadata-only: an unchanged build with the same status and the marker
  present is a silent no-op; a status flip (draft->Ready) or a missing marker
  refreshes the body without regeneration. Exclusion files are normalized to stay
  newline-terminated so appends never concatenate onto the previous entry.
- api-diff-worker.md (compiled to .lock.yml): the agent commits the generated reports
  inline (restricted to an api-diff report-tree allowlist, markdownlint-clean via
  xargs to avoid ARG_MAX limits) and publishes through gh-aw native safe outputs --
  create-pull-request / push-to-pull-request-branch open or append to the draft PR,
  update-pull-request refreshes the title and body, add-comment posts one run
  summary, mark-pull-request-as-ready-for-review flips it Ready when code-complete,
  and noop covers the silent path. Every write is constrained to the automation +
  api-diff labels and the file allowlist, and a post-step re-validates the
  agent-written state block against the host-authored target.json (marker /
  current-version / generated-at) before the run is accepted.
- Positions the working tree on the publish branch (detached at the base commit for
  a fresh diff, or the maintained PR branch for an incremental update) over an
  authenticated URL so the agent's commits land on the right ref despite the agent
  job's single-branch, credential-less checkout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let a person hand-start an API diff PR that the automation then takes over and
maintains. When no marker PR exists for a target, adopt an open PR with no
identity marker yet, resolved in priority: our exact target branch is proof of
ownership, so adopt a marker-less PR on it regardless of labels (a human may have
stripped them or bootstrapped it there); on any other branch, only adopt a PR a
human explicitly opted into automation for (both automation + api-diff) that
already introduces reports under this target's content_dir. The worker writes
the identity marker into the adopted PR's body on the first refresh and maintains
it thereafter, exactly as if it had opened the PR itself. A PR labeled only
api-diff (no automation) on another branch stays human-owned and still blocks the
automation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the worker to address reviewer feedback on the maintained draft PR on
the next scheduled run, without triggering on review events.

The agent collects the PR's submitted reviews and inline review comments itself
through GitHub integrity filtering: tools.github sets min-integrity: approved, and
features.integrity-reactions lets a maintainer promote an external contributor's
comment to approved with an endorsement reaction. Only write-access and endorsed
feedback reaches the agent, so it applies no host-side trust handling of its own.

The host setup reads the PR body's recorded generated_at as the feedback watermark
and captures this run's generated_at (its start time) in target.json, then computes
a body-free, author-agnostic wake gate that counts review activity newer than the
watermark so a caught-up build still wakes the agent for a metadata-only refresh
when new review activity exists. The agent acts on feedback created after the
watermark, folds in-scope exclusion requests into the same report update, rejects
out-of-scope requests, resolves contradictions by recency, and advances generated_at
to this run's start time so feedback is processed exactly once across runs.

Because generated_at is captured before generation begins, review activity that
arrives while a run executes carries a later timestamp and is picked up by the next
run rather than skipped.

Scope the recorded-state reads to a delimited block so human prose in the PR body
cannot shadow them. current_version, generated_at, and status are read from a fenced
yaml block the agent writes last in the body, delimited by the visible comment lines
'# api-diff:state:begin' and '# api-diff:state:end'. tracking_block in
generate.sh, and the publish guardrail, take the last begin..end range and match each
delimiter as a whole comment line, so a quoted or bulleted field line, an older state
block a maintainer pastes in as a note, or a '+ field:' line inside a diff report above
the block no longer wins the first-match extraction and cannot force a false caught-up
no-op or a bogus feedback watermark. The agent emits both delimiter lines and the
guardrail requires them, scoping its current_version/generated_at validation to the block.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeffhandley jeffhandley force-pushed the jeffhandley/api-diff-orchestrator branch from 2032529 to 7f25209 Compare July 8, 2026 06:19
Set the executable bit (100644 -> 100755) so super-linter's BASH_EXEC check
passes. The API Diff Worker workflow still invokes it via `bash <path>`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeffhandley jeffhandley marked this pull request as ready for review July 8, 2026 06:36
@jeffhandley jeffhandley requested a review from richlander July 8, 2026 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant