Skip to content

Flaky test: ConnectionTests.TestInvalidDatabase fails with Status: Unavailable #775

Description

@olavloite

Describe the bug

The unit test ConnectionTests.TestInvalidDatabase is flaky and can fail on CI runs with a gRPC Unavailable status instead of the expected NotFound status.

Error details

units (9.0.x, ubuntu-latest)	spanner-ado-net-tests	2026-06-24T11:52:28.2154838Z   Failed TestInvalidDatabase [44 ms]
units (9.0.x, ubuntu-latest)	spanner-ado-net-tests	2026-06-24T11:52:28.2155314Z   Error Message:
units (9.0.x, ubuntu-latest)	spanner-ado-net-tests	2026-06-24T11:52:28.2155982Z      Assert.That(exception.Code, Is.EqualTo(Code.NotFound))
units (9.0.x, ubuntu-latest)	spanner-ado-net-tests	2026-06-24T11:52:28.2156866Z   Expected: NotFound
units (9.0.x, ubuntu-latest)	spanner-ado-net-tests	2026-06-24T11:52:28.2157299Z   But was:  Unavailable

Stack trace

   at Google.Cloud.Spanner.DataProvider.Tests.ConnectionTests.TestInvalidDatabase() in /home/runner/work/dotnet-spanner-entity-framework/dotnet-spanner-entity-framework/spanner-ado-net/spanner-ado-net-tests/ConnectionTests.cs:line 249

Investigation & Root Cause

The flake is caused by a race condition during the rapid teardown and restart of the Go proxy server process:

  1. TestInvalidDatabase calls SpannerPool.CloseSpannerLib(), which asynchronously kills the running Go proxy server wrapper using process.Kill().
  2. This abruptly closes all active TCP connections to the in-memory C# mock server. The C# mock server needs a few milliseconds to clean up the closed channels.
  3. The test immediately opens a new connection, which spawns a new Go proxy server.
  4. The new proxy tries to connect to the C# mock server on Fixture.Port immediately. Since the mock server is still cleaning up from the abrupt teardown, it rejects the new connection, throwing Unavailable.

Proposed Fix

Update Server.Stop() to wait for the process to exit synchronously after calling Kill(), ensuring socket and handle release is complete before execution proceeds:

    public void Stop()
    {
        if (_process == null || _process.HasExited)
        {
            return;
        }
        _process.Kill();
        _process.WaitForExit(1000); // Wait up to 1 second for termination
    }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions