Per-Partition Automatic Failover: Faster detection of per-partition write region through availability strategy for writes.

[jeet1995]

Resolves #43148

Description

Problem

During per-partition automatic failover (PPAF), the backend takes up to 60s to elect a new write region for an affected server partition. During this window the SDK retries writes round-robin across regions, which is slow and causes elevated latency for customers on single-writer accounts.

Solution

This PR adds write availability strategy (hedging) for PPAF-enabled single-writer accounts. When a write to the current write region is slow or fails with a PPAF-eligible error, the SDK hedges the write to a read region via the existing ThresholdBasedAvailabilityStrategy infrastructure.

How it works

1. New config flag — COSMOS.IS_WRITE_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF (default: true) controls the feature.
2. PPAF-enforced E2E timeout for writes — mirrors the existing read availability strategy policy. Timeout = networkRequestTimeout + 1s, threshold = min(networkRequestTimeout/2, 1s), step = min(threshold/2, 500ms).
3. Region resolution — all account-level read regions (not just preferred regions) are used as hedge candidates, since PPAF failover can target any read region.
4. Hedged write routing — hedged write requests are force-routed to a target read region via routeToLocation(RegionalRoutingContext), bypassing the excluded-regions mechanism which cannot route writes to read regions on single-writer accounts.
5. Success-only failover entry creation — the PPAF ConcurrentHashMap entry is only created when the hedged write succeeds (via doOnNext callback), preventing bad regions from being persisted if the hedge fails.
6. Relaxed idempotent-write gate — PPAF provides partition-level exactly-once semantics, so write hedging is allowed even without explicit idempotentWriteRetries enabled.

Files changed (8 files, +1279 / -12)

File	Change
ClientRetryPolicy.java	Comment update + PPAF write hedging routing via writeRegionRoutingContextForPpafAvailabilityStrategy
Configs.java	New system property COSMOS.IS_WRITE_AVAILABILITY_STRATEGY_ENABLED_WITH_PPAF (default: true)
CrossRegionAvailabilityContextForRxDocumentServiceRequest.java	New volatile RegionalRoutingContext field for hedged write target region + PartitionKeyRangeWrapper for success callback
RxDocumentClientImpl.java	Core logic: evaluatePpafEnforcedE2eLatencyPolicyCfgForWrites, enableAvailabilityStrategyForWrites, region-to-routing-context map, hedged write setup with doOnNext success callback, relaxed write gates
GlobalPartitionEndpointManagerForPerPartitionAutomaticFailover.java	tryAddPartitionLevelLocationOverride routes hedged writes; tryRecordSuccessfulWriteHedge persists entry only on success
CHANGELOG.md	User-facing feature description
PerPartitionAutomaticFailoverE2ETests.java	+1054 lines: comprehensive E2E tests
IncrementalChangeFeedProcessorTest.java	Unrelated flaky test fix

---

Failover Regression Test (DR Drill)

Date: 2026-03-21 12:42-13:28 UTC | Environment: Test14 | Regions: North Central US (Write), West US (Read), East Asia (Read) | Branch: AzCosmos_WriteAvailabilityStrategyForPPAF @ 7523b1f7937 | SDK: azure-cosmos 4.79.0-beta.1 (custom JAR with write hedge fix)

Accounts

Account	Consistency	PPAF	Purpose
ppaf-strong-0321	Strong	✅ Enabled	PPAF + Strong consistency
ppaf-session-0321	Session	✅ Enabled	PPAF + Session consistency
noppaf-strong-0321	Strong	❌ Disabled	Baseline (no PPAF)

Workloads

Each account runs 2 workloads (Direct + Gateway mode), each with Create + Read + Query operations. User agent suffix identifies each workload in Kusto.

User Agent	Account	Mode	Operations
dr-ppaf-strong-0321-direct	ppaf-strong-0321	Direct	Create, Read, Query
dr-ppaf-strong-0321-gateway	ppaf-strong-0321	Gateway	Create, Read, Query
dr-ppaf-session-0321-direct	ppaf-session-0321	Direct	Create, Read, Query
dr-ppaf-session-0321-gateway	ppaf-session-0321	Gateway	Create, Read, Query
dr-noppaf-strong-0321-direct	noppaf-strong-0321	Direct	Create, Read, Query
dr-noppaf-strong-0321-gateway	noppaf-strong-0321	Gateway	Create, Read, Query

Timeline

Time (UTC)	Event
12:42:52	Drill start — 6 workloads launched
13:06:44	Quorum loss injected on all 15 partitions across 3 accounts (North Central US, 10 min duration)
~13:16:44	Quorum loss resolved
13:28:01	Drill end — all 6 workloads completed cleanly

---

Region Distribution During Failover

Observation: During QL (13:05-13:15), PPAF accounts shift writes to West US. Session consistency account shifts cleanly; Strong consistency accounts experience elevated errors due to quorum requirements. Non-PPAF baseline scatters across all regions with no directed failover.

---

Success vs Errors

Backend success rates (BackendEndRequest5M):

Account	Consistency	PPAF	Total Requests	Successes	Errors	Success Rate
ppaf-session-0321	Session	✅	12,185	12,019	166	98.64%
ppaf-strong-0321	Strong	✅	37,369	18,853	18,516	50.45%
noppaf-strong-0321	Strong	❌	128,777	22,287	106,490	17.31%

Key finding: PPAF + Session consistency achieves 98.64% success rate during quorum loss. PPAF + Strong has 50.45% (limited by quorum requirements for reads). Without PPAF, only 17.31% — a 5.7x improvement with PPAF enabled.

---

Error Breakdown During Quorum Loss Window

Account	StatusCode/Sub	Region	Count	Explanation
ppaf-session-0321	403/3	West US	6	Write forbidden during transition. Auto-retried.
ppaf-session-0321	403/3	East Asia	3	Brief write during transition. Auto-retried.
ppaf-session-0321	404/1002	NCentUS	7	ReadSessionNotAvailable. Auto-retried.
ppaf-strong-0321	410/1022	West US	10,656	PartitionKeyRangeGone — Strong requires quorum.
ppaf-strong-0321	503/1337	West US	787	Server busy during failover.
ppaf-strong-0321	403/3	West US	27	Write forbidden. Auto-retried.
noppaf-strong-0321	410/1022	West US	61,432	No directed failover — round-robin to all regions.
noppaf-strong-0321	410/1022	East Asia	36,292	Same — hits failing regions repeatedly.
noppaf-strong-0321	503/1337	West US	5,554	Server busy, no PPAF mitigation.
noppaf-strong-0321	503/1337	East Asia	2,853	Same.

---

Failover and Failback

Observation: PPAF accounts show clean failover (NCentUS drops, West US rises) and failback (NCentUS recovers post-QL). Non-PPAF baseline has no directed failover pattern.

---

PPAF vs No-PPAF Comparison

---

Verdict

Scenario	PPAF + Session	PPAF + Strong	No PPAF (baseline)	Result
Quorum Loss (write region)	98.64% success, clean failover	50.45% success (quorum-limited)	17.31% success	PPAF PASS
Failback	Clean return to NCentUS	Clean return	N/A	PASS
Write hedging	Writes shift to West US within 1 bucket	Writes shift with elevated 410s	No directed shift	PPAF PASS

PPAF write availability strategy dramatically improves resilience for Session consistency accounts (98.64% vs 17.31% baseline). Strong consistency is inherently limited by quorum requirements during partition-level failures, but PPAF still provides 3x improvement (50.45% vs 17.31%).

---

Kusto queries (cluster: cosmosdbtest.kusto.windows.net, database: Test)

Region distribution (BackendEndRequest5M):

BackendEndRequest5M
| where TIMESTAMP between (datetime(2026-03-21 12:40) .. datetime(2026-03-21 13:30))
| where GlobalDatabaseAccountName in ('ppaf-strong-0321', 'ppaf-session-0321', 'noppaf-strong-0321')
| where ResourceType == 2
| summarize Requests=sum(SampleCount) by bin(TIMESTAMP, 5m), Region, GlobalDatabaseAccountName
| render timechart

Error breakdown:

BackendEndRequest5M
| where TIMESTAMP between (datetime(2026-03-21 13:05) .. datetime(2026-03-21 13:20))
| where GlobalDatabaseAccountName in ('ppaf-strong-0321', 'ppaf-session-0321', 'noppaf-strong-0321')
| where ResourceType == 2 and StatusCode >= 400
| summarize ErrorCount=sum(SampleCount) by GlobalDatabaseAccountName, StatusCode, SubStatusCode, Region
| order by GlobalDatabaseAccountName, ErrorCount desc

---

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[jeet1995]

Investigation: nonIdempotentWriteRetryPolicy impact on PPAF write availability strategy

Finding

When PPAF write availability strategy is enabled, write hedging activates even when nonIdempotentWriteRetryPolicy is disabled (i.e., isIdempotentWriteRetriesEnabled=false). This is by design (lines 7818-7829 in getApplicableRegionsForSpeculation) — the comment states PPAF provides "exactly-once semantics for writes to failed-over partitions."

However, this creates a safety gap when both the primary and hedged writes succeed:

The Risk

Scenario	nonIdempotentWriteRetryPolicy=TRUE	nonIdempotentWriteRetryPolicy=FALSE
Tracking ID generated	✅ UUID per request	❌ None
Hedging fires (PPAF)	✅	✅ (bypasses idempotent check)
Primary + hedge both commit	409 on duplicate → read doc → verify _trackingId → return 201	Both writes persist — no dedup
ClientRetryPolicy 503 retry	Cross-region retry allowed	PPAF override: always retries (line 601-603)
Cancellation safety	Safe — _trackingId catches late commits	Unsafe — no dedup mechanism

Code Path

1. Mono.firstWithValue(monoList) fires primary + hedged requests in parallel
2. First response wins; slower subscription is cancelled
3. But cancellation is not guaranteed to prevent the write from committing — the request may already be in-flight at the backend
4. Without tracking IDs, if both writes commit:
   • Create: duplicate document (or 409 conflict with no way to detect it's "our" duplicate)
   • Replace/Patch: last-write-wins, but client only sees the first response
   • Upsert: duplicate or overwrite depending on timing

Recommendation

When applyAvailabilityStrategyForWritesForPpaf=true and isIdempotentWriteRetriesEnabled=false, the PPAF write availability strategy should ensure tracking IDs are generated for write operations. Options:

1. In evaluatePpafEnforcedE2eLatencyPolicyCfgForWrites: Return a policy config that also sets nonIdempotentWriteRetriesEnabled=true + useTrackingIds=true on the request options
2. In getApplicableRegionsForSpeculation: When the PPAF bypass fires (line 7827), also set tracking IDs on the request
3. In wrapPointOperationWithAvailabilityStrategy: When applyAvailabilityStrategyForWritesForPpaf=true, inject tracking ID generation into the callback

Option 3 is cleanest as it's scoped to the hedging path only.

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).