Add opt-in Helix work item batching#16913
Open
mmitche wants to merge 2 commits into
Open
Conversation
Add a Helix SDK batching task that groups compatible PayloadDirectory work items into staged batch payloads with per-member runners, logs, result collection, and conservative opt-in MSBuild properties. Document the public batching surface, expose send-to-helix template parameters, and cover grouping, pass-through, timeout, manifest, and shell runner behavior with unit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in, submission-time batching of Helix work items so repos sending many short PayloadDirectory/xUnit work items can amortize per-item Helix overhead. A new MSBuild task stages each member's payload under a parent batch directory, generates a shell/cmd runner that executes each command in its own subshell with per-member HELIX_WORKITEM_UPLOAD_ROOT, writes a JSON manifest, and replaces the original HelixWorkItem items with the batch items before CoreTest runs.
Changes:
- New
BatchHelixWorkItemsMSBuild task +helix-batching/HelixBatching.targetswired intoMicrosoft.DotNet.Helix.Sdk.propsvia_HelixMonoQueueTargets, withEnableHelixWorkItemBatchingand related knobs. - xUnit/xUnit v3 work item targets re-ordered (
BeforeTargets="BatchHelixWorkItems;CoreTest") so generated items are visible to batching; new params flowed througheng/common/core-templates/steps/send-to-helix.yml. - Unit tests covering grouping, pass-through, max-batch size, manifests, and POSIX shell runner generation; new sections in
Readme.mdandSendingJobsToHelix.md.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.DotNet.Helix/Sdk/BatchHelixWorkItems.cs | New MSBuild task implementing the batching, runner, and manifest generation |
| src/Microsoft.DotNet.Helix/Sdk/tools/helix-batching/HelixBatching.targets | Wires the task before CoreTest, swaps HelixWorkItem items |
| src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.props | Declares defaults, registers BatchHelixWorkItems task, hooks targets via _HelixMonoQueueTargets |
| src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets | Adds BatchHelixWorkItems to BeforeTargets so xUnit items are batched |
| src/Microsoft.DotNet.Helix/Sdk/tools/xunitv3-runner/XUnitV3Runner.targets | Same ordering change for xUnit v3 |
| eng/common/core-templates/steps/send-to-helix.yml | Surfaces batching parameters in the AzDO template |
| src/Microsoft.DotNet.Helix/Sdk.Tests/.../BatchHelixWorkItemsTests.cs | Tests for grouping, pass-through, max-size, manifest, and POSIX runner |
| src/Microsoft.DotNet.Helix/Sdk/Readme.md | Documents the new opt-in feature and HelixBatchable knob |
| Documentation/AzureDevOps/SendingJobsToHelix.md | Documents YAML usage of the batching parameters |
| </ItemGroup> | ||
| ``` | ||
|
|
||
| Set `HelixBatchable=false` on a work item or xUnit project that needs machine-level isolation. The initial batching implementation only batches simple `PayloadDirectory` items; archive/URI payloads and items with per-work-item pre/post commands are preserved unchanged. |
| { | ||
| TimeSpan targetDuration = ParseDurationOrDefault(TargetDuration, TimeSpan.FromMinutes(10), nameof(TargetDuration)); | ||
| TimeSpan timeoutPadding = ParseDurationOrDefault(TimeoutPadding, TimeSpan.FromMinutes(2), nameof(TimeoutPadding)); | ||
| int maxItemsPerBatch = Math.Max(1, MaxItemsPerBatch); |
| } | ||
| } | ||
|
|
||
| private static string EscapePosix(string value) => value.Replace("'", "'\"'\"'"); |
Comment on lines
+22
to
+24
| EnableHelixWorkItemBatching: false # optional -- batch compatible short Helix work items to reduce per-item overhead | ||
| HelixBatchTargetDuration: '00:10:00' # optional -- target aggregate duration per batch | ||
| HelixBatchMaxItems: 10 # optional -- maximum original work items per batch |
Comment on lines
+47
to
+51
| string batchRoot = Path.GetFullPath(Path.Combine(IntermediateOutputPath, "helix-work-item-batches")); | ||
| if (Directory.Exists(batchRoot)) | ||
| { | ||
| Directory.Delete(batchRoot, recursive: true); | ||
| } |
| <Project> | ||
|
|
||
| <Target Name="BatchHelixWorkItems" | ||
| Condition="'$(EnableHelixWorkItemBatching)' == 'true' and '@(HelixWorkItem)' != ''" |
- Use System.Text.Json instead of Newtonsoft.Json in batching tests - Fix Readme to clarify HelixBatchable=false applies to generated HelixWorkItems - Honor HelixBatchMinItems=1 instead of silently flooring at 2 - Remove misleading EscapePosix helper (inputs are always safe) - Expose HelixBatchTimeoutPadding/HelixBatchMinItems in send-to-helix.yml - Deterministically order BatchHelixWorkItems after all HelixWorkItem producers via AfterTargets - Replace TargetDuration/TimeoutPadding TimeSpan parsing with integer minutes; remove ParseDurationOrDefault Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
eng\\common\\dotnet.cmd test src\\Microsoft.DotNet.Helix\\Sdk.Tests\\Microsoft.DotNet.Helix.Sdk.Tests\\Microsoft.DotNet.Helix.Sdk.Tests.csproj --no-restore --verbosity minimal