Client fails to connect when a server rejects the server/discover probe with HTTP 404

[rbstp]

Describe the bug

Under the 2026-07-28 revision (SEP-2575) a client with ProtocolVersion unset probes server/discover and is documented to fall back to the initialize handshake automatically for servers that don't support it. That fallback is keyed on McpProtocolException (plus the probe timeout), so it only fires when the server answers with a JSON-RPC error.

A Streamable HTTP server that requires Mcp-Session-Id on every non-initialize POST rejects the session-less server/discover probe at the HTTP layer instead. That surfaces as HttpRequestException from EnsureSuccessStatusCodeWithResponseBodyAsync, which no catch in McpClientImpl.ConnectAsync handles, so it escapes and the connection fails outright — the fallback never runs.

StreamableHttpClientSessionTransport surfaces non-400 statuses as HttpRequestException deliberately ("404 session-not-found ... continue to surface as HttpRequestException to preserve back-compat with transport-layer behaviors"), so the gap is at the connect layer, not the transport.

Datadog's hosted MCP server (https://mcp.datadoghq.com/api/unstable/mcp-server/mcp) is a public example: it answers the probe with 404 "Invalid session ID".

To Reproduce

Steps to reproduce the behavior:

1. Connect to a Streamable HTTP server that requires a session on every non-initialize POST, with ProtocolVersion left unset:

var transport = new HttpClientTransport(new HttpClientTransportOptions
{
    Endpoint = new Uri("https://mcp.datadoghq.com/api/unstable/mcp-server/mcp"),
    TransportMode = HttpTransportMode.StreamableHttp,
    AdditionalHeaders = new Dictionary<string, string>
    {
        ["DD-API-KEY"] = "<key>",
        ["DD-APPLICATION-KEY"] = "<app-key>",
    },
});

var client = await McpClient.CreateAsync(transport, new McpClientOptions());

2. Observe the connection fail with HttpRequestException.
3. Confirm the server itself is fine; the same endpoint completes an initialize handshake over plain HTTP, negotiating 2025-11-25:

curl -sD - -X POST "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp" \
  -H "DD-API-KEY: <key>" -H "DD-APPLICATION-KEY: <app-key>" \
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2026-07-28","capabilities":{},"clientInfo":{"name":"d","version":"1"}}}'
# HTTP/2 200, mcp-session-id issued, result negotiates "protocolVersion":"2025-11-25"

4. Setting ProtocolVersion = "2025-11-25" explicitly also connects fine (30 tools), confirming only the discover probe path is affected.

Expected behavior

The client falls back to the initialize handshake and connects, as it does when the probe fails with a JSON-RPC error. A 404 on the session-less probe is evidence of an initialize-handshake server, not a fatal connection error.

Logs

System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found). Response body: Invalid session ID
   at ModelContextProtocol.HttpResponseMessageExtensions.EnsureSuccessStatusCodeWithResponseBodyAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at ModelContextProtocol.Client.StreamableHttpClientSessionTransport.SendMessageAsync(JsonRpcMessage message, CancellationToken cancellationToken)
   at ModelContextProtocol.McpSessionHandler.SendRequestAsync(JsonRpcRequest request, CancellationToken cancellationToken)
   at ModelContextProtocol.Client.McpClientImpl.SendRequestAsync(JsonRpcRequest request, CancellationToken cancellationToken)
   at ModelContextProtocol.McpSession.SendRequestAsync[TParameters,TResult](String method, TParameters parameters, ...)
   at ModelContextProtocol.Client.McpClientImpl.<ConnectAsync>g__SendDiscoverAsync|33_0(String protocolVersion, CancellationToken cancellationToken)
   at ModelContextProtocol.Client.McpClientImpl.ConnectAsync(CancellationToken cancellationToken)
   at ModelContextProtocol.Client.McpClient.CreateAsync(IClientTransport clientTransport, McpClientOptions clientOptions, ...)

Additional context

• Regression from 2.0.0. Same endpoint and credentials connect fine on 1.4.0 and 1.4.1 (30 tools) but 2.0.0 fails.
• Only affects servers that reject the probe at the HTTP layer. GitHub (api.githubcopilot.com/mcp/) and Sentry (mcp.sentry.dev) both support SEP-2575 and negotiate 2026-07-28 normally.

[rbstp]

While fixing this I hit two adjacent problems worth separating out:

1. AutoDetect has an independent channel-lifetime bug. With TransportMode.AutoDetect (the default), a 404 on the probe disposes the provisional Streamable transport and tries SSE; when the SSE GET fails, SseClientSessionTransport.SetDisconnected completes the shared message channel. Any subsequent initialize is answered by the server but its response can never be read, so the connect stalls until InitializationTimeout. Measured locally: ClientTransportClosedException after ~6s with the initialize confirmed received, versus an immediate, clear 404 without any fallback. This is why #1766 gates its fallback to the explicit Streamable HTTP transport — falling back under AutoDetect would be strictly worse than today's behavior. The real fix is for provisional SSE failure to leave the shared channel open, the way StreamableHttpClientSessionTransport already does.

2. Plain HTTP 400 is unhandled too. docs/concepts/stateless/stateless.md states that a 400 carrying anything other than a structured -32022 / -32021 / -32020 error should switch to the initialize flow. A plain-text or empty 400 fails JSON-RPC parsing, becomes HttpRequestException, and escapes ConnectAsync exactly as the 404 does — so that documented behavior is currently not implemented. It predates #1766 and is not addressed there.

Happy to open separate issues for either if that is preferable to tracking them here.