Batch oversized bulk operation audit logs by Copilot · Pull Request #5484 · microsoft/fhir-server

Copilot · 2026-04-06T16:43:50Z

Description

Bulk delete and bulk update audit logs concatenate all affected resource IDs into a single "Affected Items" string, which can exceed the 16,000 character log limit. This change splits the items into size-bounded batches, emitting multiple LogAudit calls as needed.

Created a shared BulkOperationAuditLogHelper static class in Features/Persistence/ containing the batching logic and size constants
Added MaxAuditLogSize (16000) and AuditLogOverheadSize (1000) constants
CreateAffectedItemBatches() splits items into batches where each batch string stays under the limit; items are never split mid-entry
A single oversized item (e.g., extremely long resource ID) is kept in its own batch rather than truncated
Items within a batch are comma-separated, with no leading comma on the first item
Updated DeletionService.CreateAuditLog() to use the shared helper
Updated BulkUpdateService.CreateAuditLog() to use the shared helper, replacing the previous unbounded Aggregate approach with size-safe batching

Related issues

Bug 188287

Testing

Added 8 unit tests in BulkOperationAuditLogHelperTests.cs covering: empty list, small list (single batch), large list (multi-batch), boundary sizing, include-tag preservation, no leading comma per batch, single oversized item, and no-data-loss across batches.

Added integration tests verifying the audit logging helper is invoked by both services:

DeletionServiceAuditLogTests.cs: GivenBulkHardDelete_WhenResourcesAreDeleted_ThenAuditLoggerIsCalledWithBatchedAffectedItems — exercises the full DeleteMultipleAsync hard-delete flow and asserts _auditLogger.LogAudit is called with "Affected Items" in additionalProperties.
BulkUpdateServiceTests.cs: UpdateMultipleAsync_WhenResultsReturned_ThenAuditLoggerIsCalledWithBatchedAffectedItems — exercises the UpdateMultipleAsync flow and asserts the same.

Both integration tests use a polling pattern (instead of hard-coded delays) to deterministically wait for Task.Run-based audit logging to complete.

FHIR Team Checklist

Update the title of the PR to be succinct and less than 65 characters
Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
Tag the PR with Schema Version backward compatible or Schema Version backward incompatible or Schema Version unchanged if this adds or updates Sql script which is/is not backward compatible with the code.
When changing or adding behavior, if your code modifies the system design or changes design assumptions, please create and include an ADR.
CI is green before merge
Review squash-merge requirements

Semver Change (docs)

Patch

…ay within 16,000 char limit Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/20d0d51f-b8cc-4da8-9995-a01c7b432716 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

…eateAffectedItemBatches Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/20d0d51f-b8cc-4da8-9995-a01c7b432716 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

LTA-Thinking · 2026-04-06T18:02:36Z

src/Microsoft.Health.Fhir.Shared.Core/Features/Resources/Delete/DeletionService.cs

+
+            foreach (var item in items)
+            {
+                string itemString = $", {(item.included ? "[Include] " : string.Empty)}{item.resourceType}/{item.resourceId}";


@copilot Please remove the comma from the first item added to a batch.

Done in d145517. The first item in each batch no longer has a leading comma — commas are now only used as separators between items within a batch.

Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/5263344e-c535-4e76-b6ea-94c6ffa85136 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

LTA-Thinking · 2026-04-06T18:09:05Z

/azp run

azure-pipelines · 2026-04-06T18:09:15Z

Azure Pipelines successfully started running 1 pipeline(s).

codecov-commenter · 2026-04-06T19:22:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.95%. Comparing base (ea948f8) to head (d145517).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5484      +/-   ##
==========================================
+ Coverage   76.91%   76.95%   +0.04%     
==========================================
  Files         978      978              
  Lines       35680    35694      +14     
  Branches     5356     5359       +3     
==========================================
+ Hits        27443    27470      +27     
+ Misses       6903     6892      -11     
+ Partials     1334     1332       -2

see 12 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

apurvabhaleMS · 2026-04-06T20:23:11Z

@LTA-Thinking do we need to do this for Bulk update as well?

LTA-Thinking · 2026-04-06T20:56:02Z

@copilot Please change the update service to use the same audit log method as bulk delete. The create audit log method can be moved to a new static class that the update service and delete service can reference.

…ass; update BulkUpdateService to use batching Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/a1a789ae-7398-4de7-89c9-188742ff4760 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Copilot · 2026-04-06T21:04:32Z

@copilot Please change the update service to use the same audit log method as bulk delete. The create audit log method can be moved to a new static class that the update service and delete service can...

Done in c1d4731. Extracted CreateAffectedItemBatches() and the size constants into a new BulkOperationAuditLogHelper static class under Features/Persistence/. Both DeletionService and BulkUpdateService now reference it. The BulkUpdateService.CreateAuditLog method now uses batching (replacing the old Aggregate approach), so oversized audit logs are handled for bulk update as well.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

microsofthealthoss.pkgs.visualstudio.com
- Triggering command: /usr/bin/dotnet dotnet build src/Microsoft.Health.Fhir.R4.Core/Microsoft.Health.Fhir.R4.Core.csproj --no-restore (dns block)
- Triggering command: /usr/bin/dotnet dotnet restore src/Microsoft.Health.Fhir.R4.Core/Microsoft.Health.Fhir.R4.Core.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

LTA-Thinking · 2026-04-06T21:08:32Z

/azp run

azure-pipelines · 2026-04-06T21:08:49Z

Azure Pipelines successfully started running 1 pipeline(s).

LTA-Thinking · 2026-04-06T21:18:27Z

@copilot Please add unit tests for the deletion service that include a test showing the audit logging helper is being called. Also, update the bulk update service's unit tests to add one showing the audit logging helping is being called.

…vice Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/eb92acd7-9bb1-45a0-9e08-1a937fe7e92e Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

…rtions Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/eb92acd7-9bb1-45a0-9e08-1a937fe7e92e Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Copilot · 2026-04-06T21:33:08Z

@copilot Please add unit tests for the deletion service that include a test showing the audit logging helper is being called. Also, update the bulk update service's unit tests to add one showing the a...

Done in dc278bd. Added DeletionServiceAuditLogTests.cs with a test GivenBulkHardDelete_WhenResourcesAreDeleted_ThenAuditLoggerIsCalledWithBatchedAffectedItems that exercises the full DeleteMultipleAsync flow and verifies _auditLogger.LogAudit is called with "Affected Items" in additionalProperties. Also added UpdateMultipleAsync_WhenResultsReturned_ThenAuditLoggerIsCalledWithBatchedAffectedItems to BulkUpdateServiceTests.cs with the same verification for the update path. Both tests use a polling pattern instead of hard-coded delays to wait for the Task.Run-based audit logging.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

microsofthealthoss.pkgs.visualstudio.com
- Triggering command: /usr/bin/dotnet dotnet build src/Microsoft.Health.Fhir.R4.Core.UnitTests/Microsoft.Health.Fhir.R4.Core.UnitTests.csproj (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

LTA-Thinking · 2026-04-06T21:36:50Z

/azp run

azure-pipelines · 2026-04-06T21:37:08Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot AI and others added 2 commits April 6, 2026 16:32

Split large audit logs in DeletionService into multiple batches to st…

b8be09e

…ay within 16,000 char limit Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/20d0d51f-b8cc-4da8-9995-a01c7b432716 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Address code review: use StringBuilder and string interpolation in Cr…

c766847

…eateAffectedItemBatches Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/20d0d51f-b8cc-4da8-9995-a01c7b432716 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Copilot AI assigned Copilot and LTA-Thinking Apr 6, 2026

Copilot created this pull request from a session on behalf of LTA-Thinking April 6, 2026 16:43 View session

LTA-Thinking reviewed Apr 6, 2026

View reviewed changes

Copilot started work on behalf of LTA-Thinking April 6, 2026 18:02 View session

Remove leading comma from first item in each audit log batch

d145517

Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/5263344e-c535-4e76-b6ea-94c6ffa85136 Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Copilot finished work on behalf of LTA-Thinking April 6, 2026 18:05

Copilot AI requested a review from LTA-Thinking April 6, 2026 18:05

LTA-Thinking marked this pull request as ready for review April 6, 2026 18:08

LTA-Thinking requested a review from a team as a code owner April 6, 2026 18:08

LTA-Thinking added this to the FY26\Q4\2Wk\2Wk21 milestone Apr 6, 2026

LTA-Thinking approved these changes Apr 6, 2026

View reviewed changes

Copilot started work on behalf of LTA-Thinking April 6, 2026 20:56 View session

Copilot AI changed the title ~~Split oversized bulk delete audit logs into batches~~ Batch oversized bulk operation audit logs Apr 6, 2026

Copilot AI requested a review from LTA-Thinking April 6, 2026 21:05

Copilot finished work on behalf of LTA-Thinking April 6, 2026 21:05

LTA-Thinking approved these changes Apr 6, 2026

View reviewed changes

Copilot started work on behalf of LTA-Thinking April 6, 2026 21:18 View session

Copilot AI and others added 2 commits April 6, 2026 21:31

Add audit log integration tests for DeletionService and BulkUpdateSer…

612dd56

…vice Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/eb92acd7-9bb1-45a0-9e08-1a937fe7e92e Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Replace hard-coded Task.Delay with polling pattern for audit log asse…

dc278bd

…rtions Agent-Logs-Url: https://github.com/microsoft/fhir-server/sessions/eb92acd7-9bb1-45a0-9e08-1a937fe7e92e Co-authored-by: LTA-Thinking <17463554+LTA-Thinking@users.noreply.github.com>

Copilot finished work on behalf of LTA-Thinking April 6, 2026 21:34

Copilot AI requested a review from LTA-Thinking April 6, 2026 21:34

Conversation

Copilot AI commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues

Testing

FHIR Team Checklist

Semver Change (docs)

Uh oh!

LTA-Thinking Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

LTA-Thinking commented Apr 6, 2026

Uh oh!

azure-pipelines bot commented Apr 6, 2026

Uh oh!

codecov-commenter commented Apr 6, 2026

Codecov Report

Uh oh!

apurvabhaleMS commented Apr 6, 2026

Uh oh!

LTA-Thinking commented Apr 6, 2026

Uh oh!

Copilot AI commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

LTA-Thinking commented Apr 6, 2026

Uh oh!

azure-pipelines bot commented Apr 6, 2026

Uh oh!

LTA-Thinking commented Apr 6, 2026

Uh oh!

Copilot AI commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

LTA-Thinking commented Apr 6, 2026

Uh oh!

azure-pipelines bot commented Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Apr 6, 2026 •

edited

Loading

Copilot AI commented Apr 6, 2026 •

edited

Loading

Copilot AI commented Apr 6, 2026 •

edited

Loading