PR #6: Bug - Apply DefaultDeadline unconditionally in BuildClientSettings to fix silent timeout bypass

[AlexanderJohnston]

Problem

When ESDB experiences transient I/O pressure (chunk caching/uncaching, storage writer queue backup), gRPC calls from Totem can exceed the configured DefaultDeadline and return DeadlineExceeded. This permanently kills the affected flow — there is no retry, no backoff, and no recovery without manual intervention.

BuildClientSettings in EventStoreServiceExtensions has two paths depending on whether options.ConnectionString is set:

static EventStoreClientSettings BuildClientSettings(EventStoreTimelineOptions options, IServiceProvider provider)
{
    if(!string.IsNullOrEmpty(options.ConnectionString))
    {
        var settings = EventStoreClientSettings.Create(options.ConnectionString);
        settings.LoggerFactory = provider.GetService<ILoggerFactory>();
        return settings;  // ← DefaultDeadline is NEVER set
    }

    // ... structured path ...
    s.DefaultDeadline = options.Connection.Timeout;  // ← only reached here
    return s;
}

When the ConnectionString early-return path is taken, DefaultDeadline is never assigned. The gRPC client is created with no per-call deadline, falling back to whatever the EventStore .NET client's internal default happens to be (no deadline by default in gRPC — calls can wait indefinitely, or the client may impose its own).

The ConnectionString property defaults to null, but if any configuration source populates it — an environment variable, a Docker/Kubernetes secret, a hosting platform injecting ConnectionStrings__EventStore, or any other source in Microsoft's configuration merge order (JSON → env vars → command line) — the structured options in appsettings.json are silently bypassed. The operator's 15-minute timeout configuration has no effect.

Even if the ConnectionString path is not the active cause of the 10s deadline in this specific incident, the bug is real: any operator who switches to connection-string-based configuration will silently lose all timeout protection. LoggerFactory had the same issue — it was set redundantly in both paths, but DefaultDeadline was only set in one.

Fix

Restructure BuildClientSettings so both paths converge before applying shared settings:

static EventStoreClientSettings BuildClientSettings(EventStoreTimelineOptions options, IServiceProvider provider)
{
    EventStoreClientSettings s;

    if(!string.IsNullOrEmpty(options.ConnectionString))
    {
        s = EventStoreClientSettings.Create(options.ConnectionString);
    }
    else
    {
        // Omit the credentials when insecure mode is true

        var tls = options.Server.Insecure ? "tls=false" : "";
        var includeCredentials = !options.Server.Insecure && !string.IsNullOrEmpty(options.Connection.Username);

        var connStr = includeCredentials
            ? $"esdb://{options.Connection.Username}:{options.Connection.Password}@{options.Server.Name}:{options.Server.Port}?{tls}"
            : $"esdb://{options.Server.Name}:{options.Server.Port}?{tls}";

        s = EventStoreClientSettings.Create(connStr);
    }

    s.LoggerFactory = provider.GetService<ILoggerFactory>();
    s.DefaultDeadline = options.Connection.Timeout;
    return s;
}

LoggerFactory and DefaultDeadline are now assigned exactly once, unconditionally, after both paths produce an EventStoreClientSettings instance.

What this preserves

Behaviour	Before	After
Structured path (no ConnectionString)	DefaultDeadline set from options.Connection.Timeout	✅ Identical
ConnectionString path	DefaultDeadline not set — silent omission	✅ Now set from options.Connection.Timeout
LoggerFactory assignment	Set in both paths (redundant)	Set once after convergence
Connection string parsing	EventStoreClientSettings.Create(connStr)	✅ Identical
Structured connection building	Username/password/TLS logic	✅ Identical — just indented into else block

What this changes

• ConnectionString path now gets a DefaultDeadline. Previously it had none. If the connection string itself contains a timeout parameter, DefaultDeadline sets the per-call default but gRPC per-call deadlines explicitly passed to individual operations still take precedence. No conflict.
• Code structure. The early return is replaced with an if/else that converges. Three fewer lines total. No new dependencies.

Scope

One method in one file. No new packages, no behavioural changes to the structured path, no changes to flow runtime logic.