Distributed Cache Sync and Atomic Search Param Operations#5479
Distributed Cache Sync and Atomic Search Param Operations#5479SergeyGaluzo wants to merge 98 commits intomainfrom
Conversation
…ed procedure - Added new schema version V104 to SchemaVersion.cs and updated SchemaVersionConstants.cs to reflect the new maximum version. - Enhanced the MergeSearchParams stored procedure to include additional parameters for resource change capture and transaction handling. - Updated SqlServerSearchParameterStatusDataStore to handle new parameters based on schema version. - Modified SqlServerFhirDataStore to conditionally use MergeSearchParams based on pending search parameter updates. - Updated project file to set LatestSchemaVersion to 104. - Refactored integration tests to accommodate changes in search parameter handling and ensure proper synchronization.
…s and improved URL resolution
…and remove unused methods
…ns and data store
…ency control and ensure proper transaction management
…n SqlServerFhirDataStore to differentiate between transaction and non-transaction operations
…enhancing transaction handling and cache management
- Added new schema version V108 to SchemaVersion.cs. - Updated SchemaVersionConstants.cs to set Max schema version to 108. - Modified SqlServerSearchParameterStatusDataStore.cs to check for schema version 108 for enabling resource change capture. - Updated project file to reflect the latest schema version as 108.
…d search parameter operations
…trada/atomicsearchparameteroperations
- Updated SchemaVersion.cs to include V108. - Adjusted SchemaVersionConstants.cs to set Max to V108. - Modified SqlServerSearchParameterStatusDataStore.cs to check for schema version 108. - Updated project file to reflect the latest schema version as 108.
…in bulk delete operations and centralize status updates to SearchParametersOperations class - Added ISearchParameterOperations to BulkDeleteProcessingJob and BulkDeleteController to check for active reindex jobs. - Updated BulkDeleteProcessingJob to ensure no active reindex job before executing delete operations. - Enhanced BulkDeleteController to handle conflicts when reindexing is in progress. - Introduced unit tests to validate behavior during reindexing scenarios in BulkDelete and SearchParameterStateUpdateHandler. - Refactored SearchParameterValidator to utilize ISearchParameterOperations for reindex job checks.
…and improve logging for unresolved URLs
...Health.Fhir.Shared.Core.UnitTests/Features/Operations/Reindex/ReindexOrchestratorJobTests.cs
Dismissed
Show dismissed
Hide dismissed
...t.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
Fixed
Show fixed
Hide fixed
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5479 +/- ##
==========================================
+ Coverage 76.86% 77.01% +0.14%
==========================================
Files 976 979 +3
Lines 35598 35893 +295
Branches 5353 5422 +69
==========================================
+ Hits 27364 27643 +279
+ Misses 6905 6896 -9
- Partials 1329 1354 +25 🚀 New features to boost your workflow:
|
...t.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
Fixed
Show fixed
Hide fixed
...t.Health.Fhir.SqlServer/Features/Storage/Registry/SqlServerSearchParameterStatusDataStore.cs
Fixed
Show fixed
Hide fixed
This reverts commit 19ede1e.
src/Microsoft.Health.Fhir.Core/Features/Operations/Reindex/ReindexOrchestratorJob.cs
Dismissed
Show dismissed
Hide dismissed
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| foreach (var hostName in activeHosts) | ||
| { | ||
| // There is always a time gap of several milliseconds to several seconds between search indexes generation and surrogate ids assignment. | ||
| // We need to make sure that during this time search param cache does not change. | ||
| // Hence we check that last updated is identical on last to update events. | ||
| if (eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated) | ||
| { | ||
| updatedHosts.Add(hostName, value[0].lastUpdated); | ||
| } | ||
| } |
Check notice
Code scanning / CodeQL
Missed opportunity to use Where Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 17 hours ago
In general, to fix this kind of issue, we move the filtering condition from inside the loop into a LINQ .Where(...) clause applied to the sequence we’re iterating, optionally projecting into an anonymous type if we need both the original element and extra computed data. This reduces nesting and makes it explicit that we operate only on a subset of the original collection.
Here, instead of looping over all activeHosts and then checking eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated, we can construct an intermediate enumerable that pairs each hostName with its corresponding value (when present), filter that sequence by the same condition, and then populate updatedHosts from it. A good way, without changing functionality, is:
- Keep
activeHostsas is. - Build an enumerable like
activeHosts.Select(hostName => new { hostName, value = eventsByHosts.TryGetValue(...) ? value : null }). - Apply
.Where(x => x.value != null && x.value.Count == 2 && x.value[0].lastUpdated == x.value[1].lastUpdated). - Inside a
foreachover this filtered sequence, callupdatedHosts.Add(hostName, lastUpdated).
This keeps a single dictionary lookup per host and preserves the semantics of only adding to updatedHosts when the condition holds. The change is local to the loop around lines 286–294 in SqlServerSearchParameterStatusDataStore.cs; all required namespaces (System.Linq) are already imported at the top of the file, so no new imports or methods are needed.
| @@ -283,15 +283,18 @@ | ||
| .Select(g => new { g.Key, Value = g.OrderByDescending(_ => _.eventDate).Take(2).ToList() }) | ||
| .ToDictionary(_ => _.Key, _ => _.Value); | ||
| var updatedHosts = new Dictionary<string, string>(); | ||
| foreach (var hostName in activeHosts) | ||
| foreach (var item in activeHosts | ||
| .Select(hostName => | ||
| { | ||
| eventsByHosts.TryGetValue(hostName, out var value); | ||
| return new { hostName, value }; | ||
| }) | ||
| .Where(x => x.value != null && x.value.Count == 2 && x.value[0].lastUpdated == x.value[1].lastUpdated)) | ||
| { | ||
| // There is always a time gap of several milliseconds to several seconds between search indexes generation and surrogate ids assignment. | ||
| // We need to make sure that during this time search param cache does not change. | ||
| // Hence we check that last updated is identical on last to update events. | ||
| if (eventsByHosts.TryGetValue(hostName, out var value) && value.Count == 2 && value[0].lastUpdated == value[1].lastUpdated) | ||
| { | ||
| updatedHosts.Add(hostName, value[0].lastUpdated); | ||
| } | ||
| updatedHosts.Add(item.hostName, item.value[0].lastUpdated); | ||
| } | ||
|
|
||
| var maxLastUpdated = updatedHosts.Values.Max(_ => _); |
No description provided.