Skip to content

Distributed Cache Sync and Atomic Search Param Operations#5479

Open
SergeyGaluzo wants to merge 98 commits intomainfrom
users/sergal/atomic-plus
Open

Distributed Cache Sync and Atomic Search Param Operations#5479
SergeyGaluzo wants to merge 98 commits intomainfrom
users/sergal/atomic-plus

Conversation

@SergeyGaluzo
Copy link
Copy Markdown
Contributor

No description provided.

…ed procedure

- Added new schema version V104 to SchemaVersion.cs and updated SchemaVersionConstants.cs to reflect the new maximum version.
- Enhanced the MergeSearchParams stored procedure to include additional parameters for resource change capture and transaction handling.
- Updated SqlServerSearchParameterStatusDataStore to handle new parameters based on schema version.
- Modified SqlServerFhirDataStore to conditionally use MergeSearchParams based on pending search parameter updates.
- Updated project file to set LatestSchemaVersion to 104.
- Refactored integration tests to accommodate changes in search parameter handling and ensure proper synchronization.
…ency control and ensure proper transaction management
…n SqlServerFhirDataStore to differentiate between transaction and non-transaction operations
…enhancing transaction handling and cache management
- Added new schema version V108 to SchemaVersion.cs.
- Updated SchemaVersionConstants.cs to set Max schema version to 108.
- Modified SqlServerSearchParameterStatusDataStore.cs to check for schema version 108 for enabling resource change capture.
- Updated project file to reflect the latest schema version as 108.
- Updated SchemaVersion.cs to include V108.
- Adjusted SchemaVersionConstants.cs to set Max to V108.
- Modified SqlServerSearchParameterStatusDataStore.cs to check for schema version 108.
- Updated project file to reflect the latest schema version as 108.
…in bulk delete operations and centralize status updates to SearchParametersOperations class

- Added ISearchParameterOperations to BulkDeleteProcessingJob and BulkDeleteController to check for active reindex jobs.
- Updated BulkDeleteProcessingJob to ensure no active reindex job before executing delete operations.
- Enhanced BulkDeleteController to handle conflicts when reindexing is in progress.
- Introduced unit tests to validate behavior during reindexing scenarios in BulkDelete and SearchParameterStateUpdateHandler.
- Refactored SearchParameterValidator to utilize ISearchParameterOperations for reindex job checks.
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.01%. Comparing base (2b9e554) to head (cdba4b9).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5479      +/-   ##
==========================================
+ Coverage   76.86%   77.01%   +0.14%     
==========================================
  Files         976      979       +3     
  Lines       35598    35893     +295     
  Branches     5353     5422      +69     
==========================================
+ Hits        27364    27643     +279     
+ Misses       6905     6896       -9     
- Partials     1329     1354      +25     

see 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@SergeyGaluzo SergeyGaluzo changed the title Distributed cache sync Distributed Cache Sync and Atomic Search Param Operations Apr 3, 2026
@SergeyGaluzo
Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Comment on lines +286 to +295
foreach (var hostName in activeHosts)
{
// There is always a time gap of several milliseconds to several seconds between search indexes generation and surrogate ids assignment.
// We need to make sure that during this time search param cache does not change.
// Hence we check that last updated is identical on last to update events.
if (eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated)
{
updatedHosts.Add(hostName, value[0].lastUpdated);
}
}

Check notice

Code scanning / CodeQL

Missed opportunity to use Where Note

This foreach loop
implicitly filters its target sequence
- consider filtering the sequence explicitly using '.Where(...)'.

Copilot Autofix

AI about 17 hours ago

In general, to fix this kind of issue, we move the filtering condition from inside the loop into a LINQ .Where(...) clause applied to the sequence we’re iterating, optionally projecting into an anonymous type if we need both the original element and extra computed data. This reduces nesting and makes it explicit that we operate only on a subset of the original collection.

Here, instead of looping over all activeHosts and then checking eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated, we can construct an intermediate enumerable that pairs each hostName with its corresponding value (when present), filter that sequence by the same condition, and then populate updatedHosts from it. A good way, without changing functionality, is:

  • Keep activeHosts as is.
  • Build an enumerable like activeHosts.Select(hostName => new { hostName, value = eventsByHosts.TryGetValue(...) ? value : null }).
  • Apply .Where(x => x.value != null && x.value.Count == 2 && x.value[0].lastUpdated == x.value[1].lastUpdated).
  • Inside a foreach over this filtered sequence, call updatedHosts.Add(hostName, lastUpdated).

This keeps a single dictionary lookup per host and preserves the semantics of only adding to updatedHosts when the condition holds. The change is local to the loop around lines 286–294 in SqlServerSearchParameterStatusDataStore.cs; all required namespaces (System.Linq) are already imported at the top of the file, so no new imports or methods are needed.

Suggested changeset 1
src/Microsoft.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/src/Microsoft.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs b/src/Microsoft.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
--- a/src/Microsoft.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
+++ b/src/Microsoft.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
@@ -283,15 +283,18 @@
                                        .Select(g => new { g.Key, Value = g.OrderByDescending(_ => _.eventDate).Take(2).ToList() })
                                        .ToDictionary(_ => _.Key, _ => _.Value);
             var updatedHosts = new Dictionary<string, string>();
-            foreach (var hostName in activeHosts)
+            foreach (var item in activeHosts
+                .Select(hostName =>
+                {
+                    eventsByHosts.TryGetValue(hostName, out var value);
+                    return new { hostName, value };
+                })
+                .Where(x => x.value != null && x.value.Count == 2 && x.value[0].lastUpdated == x.value[1].lastUpdated))
             {
                 // There is always a time gap of several milliseconds to several seconds between search indexes generation and surrogate ids assignment.
                 // We need to make sure that during this time search param cache does not change.
                 // Hence we check that last updated is identical on last to update events.
-                if (eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated)
-                {
-                    updatedHosts.Add(hostName, value[0].lastUpdated);
-                }
+                updatedHosts.Add(item.hostName, item.value[0].lastUpdated);
             }
 
             var maxLastUpdated = updatedHosts.Values.Max(_ => _);
EOF
@@ -283,15 +283,18 @@
.Select(g => new { g.Key, Value = g.OrderByDescending(_ => _.eventDate).Take(2).ToList() })
.ToDictionary(_ => _.Key, _ => _.Value);
var updatedHosts = new Dictionary<string, string>();
foreach (var hostName in activeHosts)
foreach (var item in activeHosts
.Select(hostName =>
{
eventsByHosts.TryGetValue(hostName, out var value);
return new { hostName, value };
})
.Where(x => x.value != null && x.value.Count == 2 && x.value[0].lastUpdated == x.value[1].lastUpdated))
{
// There is always a time gap of several milliseconds to several seconds between search indexes generation and surrogate ids assignment.
// We need to make sure that during this time search param cache does not change.
// Hence we check that last updated is identical on last to update events.
if (eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated)
{
updatedHosts.Add(hostName, value[0].lastUpdated);
}
updatedHosts.Add(item.hostName, item.value[0].lastUpdated);
}

var maxLastUpdated = updatedHosts.Values.Max(_ => _);
Copilot is powered by AI and may make mistakes. Always verify output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

SQL Scripts If SQL scripts are added to the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants