Skip to content

fix(spanner): recover inline-begin read-only transaction after failed first statement#3

Open
fornwall wants to merge 1 commit into
mainfrom
fix/p0-3-readonly-inline-begin-poisoned
Open

fix(spanner): recover inline-begin read-only transaction after failed first statement#3
fornwall wants to merge 1 commit into
mainfrom
fix/p0-3-readonly-inline-begin-poisoned

Conversation

@fornwall

@fornwall fornwall commented Jul 18, 2026

Copy link
Copy Markdown
Owner

A MultiUseReadOnlyTransaction using Options.BeginTransactionOption.INLINE (added in googleapis#13233) is permanently poisoned when the statement that carries the inline BeginTransaction fails before a transaction is returned.

AbstractReadContext.MultiUseReadOnlyTransaction.transactionIdFuture is created once by the first statement and never reset. If that begin-carrying statement fails (bad SQL, non-retryable RPC error, no transaction returned in the response metadata, or its ResultSet being closed before the transaction id arrived), failTransactionIdFuture sets the exception on the future but leaves it in place. Consequences:

  • Every subsequent statement on the transaction waits on the failed future and rethrows the stale error of the first statement, forever. The transaction object is unusable, even though no transaction was ever begun on the backend.
  • Concurrently waiting statements fail with an error that belongs to an unrelated statement (e.g. a perfectly valid read fails with INVALID_ARGUMENT: bad query).

The EXPLICIT begin path (the default) does not have this problem: initTransaction() simply re-attempts the BeginTransaction RPC on the next statement. The read/write inline-begin path recovers, too, by simulating an ABORTED error so TransactionRunner retries the whole transaction — but the read-only path has no runner and therefore no retry mechanism at all.

Note: the poisoning can only occur when no transaction id was ever received, i.e. no data was read under any transaction, so re-attempting the begin cannot violate read-timestamp consistency.

Fix

In AbstractReadContext.MultiUseReadOnlyTransaction:

  1. failTransactionIdFuture now resets transactionIdFuture to null (under txnLock) after setting the exception, so the next statement attempts a fresh inline BeginTransaction — mirroring the explicit-begin recovery behavior.
  2. getTransactionSelector now retries its selector logic in a loop (bounded by the existing 60s overall deadline) when waiting on the future fails with ExecutionException: a concurrently waiting statement takes over the inline begin itself (or waits for another statement that already has) instead of failing with the error of the unrelated first statement. The error still propagates to the failed statement itself. Read-only transactions take no locks, so unlike the read/write path there is no need to abort/retry the entire transaction.

Tests

SessionImplTest:

  • multiUseReadOnlyTransactionInlineBeginRetriesBeginAfterFailedFirstStatement (rewritten from multiUseReadOnlyTransactionInlineBeginFirstQueryErrorPropagates, which asserted the poisoned behavior): first query fails with INVALID_ARGUMENT and propagates; the next statement issues a new request with TransactionSelector.begin, and the statement after that uses the returned transaction id.
  • multiUseReadOnlyTransactionInlineBeginRecoversWhenNoTransactionIsReturned: the begin-carrying statement gets a response without transaction metadata (FAILED_PRECONDITION); the next statement re-attempts the inline begin instead of rethrowing.
  • multiUseReadOnlyTransactionInlineBeginConcurrentStatementTakesOverFailedBegin: statement B is blocked waiting for statement A's inline begin; A fails; B takes over with its own TransactionSelector.begin request and succeeds, while A's caller gets A's actual error.

All existing inline-begin tests, SessionImplTest, and the full google-cloud-spanner unit test suite pass.

@fornwall

Copy link
Copy Markdown
Owner Author

Verified end-to-end against the Spanner emulator: https://github.com/fornwall/spanner-java-issues/tree/fix/p0-3-readonly-inline-begin-poisoned

The reproducer test inlineBeginReadOnlyTransactionIsNotPoisonedByFailedFirstStatement fails against main (2f915ab) — a valid SELECT COUNT(*) after a failed first statement rethrows the first statement's stale INVALID_ARGUMENT: Unrecognized name: BadColumn error — and passes against this PR's branch.

… first statement

The transactionIdFuture of a MultiUseReadOnlyTransaction with
BeginTransactionOption.INLINE was created once and never reset. If the
statement carrying the inline BeginTransaction failed (or its ResultSet
was closed before a transaction id was returned), every subsequent
statement on the transaction rethrew the stale error of that first
statement forever, and concurrently waiting statements failed with an
error belonging to an unrelated statement.

Reset transactionIdFuture when it is failed, and retry the selector
loop in getTransactionSelector, so the next (or a concurrently waiting)
statement attempts a new inline BeginTransaction, matching the recovery
behavior of the explicit BeginTransaction path.

Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
@fornwall
fornwall force-pushed the fix/p0-3-readonly-inline-begin-poisoned branch from d1589dd to 958ac11 Compare July 18, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant