Increase api-server HTTP timeouts for devnet (hosted Canton)

[sadiq1971]

Problem

On the deployed devnet, registration and transfer requests time out at the client (~15s) but the underlying Canton operation eventually succeeds — balances update on refresh. Local docker is unaffected because everything is sub-100ms there.

Root cause

http.Server.WriteTimeout = 15s (from `server.write_timeout`) is the outermost deadline applied at the TCP socket — fires before the handler can finish a Canton-backed operation on a WAN-hosted node:

• Configured at `pkg/app/http/server.go:40` ← `pkg/app/http/config.go:10` (default `15s`)
• Devnet/mainnet yamls leave it at 15s:
  • `pkg/config/defaults/config.api-server.local-devnet.yaml:6`
  • `pkg/config/defaults/config.api-server.mainnet.yaml:5`

The larger inner timeouts never get a chance to apply:

• chi `middleware.Timeout(60s)` at `pkg/app/api/server.go:344`
• chi `middleware.Timeout(30s)` for `/eth` at `pkg/ethrpc/service/http.go:35` (from `eth_rpc.request_timeout`)

Slow paths on hosted Canton:

• `POST /register` → `AllocateExternalPartyWithSignature` (topology submit + propagation) + `CreateFingerprintMapping` (SubmitAndWait) + RDS write
• `eth_sendRawTransaction` → registry HTTP (USDCx) + `PrepareSubmission` + `ExecuteSubmission` (SubmitAndWait)

Both routinely exceed 15s when Canton is remote.

Fix

Raise `server.write_timeout` (and align `server.read_timeout` / chi timeout) in devnet + mainnet configs to a value that covers realistic Canton latency (suggest 60s). Verify ordering: `http.Server.WriteTimeout` ≥ `chi.middleware.Timeout` ≥ `eth_rpc.request_timeout` so the inner deadlines actually have effect.

Acceptance

• Devnet registration and USDCx transfer complete without client-side timeout under normal Canton latency
• Timeout layering documented in `pkg/app/http/config.go`

Note

This is a stopgap. The architectural fix for `eth_sendRawTransaction` is tracked separately (async submission via the existing mempool/miner infrastructure).

[sadiq1971]

Additional manifestation surfaced via the dApp's Activity tab

Confirmed the timeout bug end-to-end on devnet today. Captured eth_sendRawTransaction failing on the deployed v0.4.1 build (devnet still hasn't picked up #280):

ERROR  service/log.go:205  SendRawTransaction failed
  service: EthService
  tx_hash: 0x0000...0000
  duration: 14.900470773s
  error: canton transfer failed: execute submission: rpc error: code = Canceled desc = context canceled

Canton actually committed the transfer (balances reflect it on refresh), but the dApp's Activity tab stays blank for the transfer. Once #280 ships to devnet, most transfers will complete inside the new 60s envelope — but there's a residual code-level bug the config bump alone doesn't fix:

Residual bug: status updates use the canceled HTTP context

pkg/ethrpc/service/service.go:244,252

transferErr := erc20.TransferFrom(ctx, txHash.Hex(), from, toAddr, *amount)
if transferErr != nil {
    if updateErr := s.store.FailMempoolEntry(ctx, txHash.Bytes(), transferErr.Error()); updateErr != nil {
        // Non-fatal: the miner's reconciler will handle stuck pending entries.
        _ = updateErr
    }
    return common.Hash{}, transferErr
}
if updateErr := s.store.CompleteMempoolEntry(ctx, txHash.Bytes()); updateErr != nil {
    // Non-fatal: the miner's reconciler will handle stuck pending entries.
    _ = updateErr
}

When the HTTP request context fires (timeout, or any other cancel), transferErr is context canceled — and the very next line tries to write FailMempoolEntry using the same canceled ctx, so the DB update is rejected. The _ = updateErr swallows it. The mempool entry stays at status='pending' forever. The miner only claims WHERE status='completed' (pg/pg.go:88), so it skips this entry permanently → no evm_logs row → Activity tab stays blank for that transfer.

The comments at lines 245 and 253 ("the miner's reconciler will handle stuck pending entries") are aspirational — there is no reconciler today (see #278).

Minimal proposed follow-up (this issue's scope)

Detach the status write from the request context so it survives cancellation:

func (s *ethService) recordMempoolOutcome(txHash []byte, transferErr error) {
    // Detached, time-bounded context so the status write survives the
    // HTTP request's cancellation. Canton submissions are asynchronous
    // on its side, so even after our `ctx` is canceled the transfer may
    // have committed — we still need to record the outcome locally.
    ctx, cancel := context.WithTimeout(context.Background(), mempoolStatusUpdateTimeout)
    defer cancel()

    if transferErr != nil && errors.Is(transferErr, context.Canceled) {
        // We genuinely don't know whether Canton committed. Leave the
        // entry 'pending' so the reconciler (#278) can resolve it
        // against Canton later. Marking 'failed' here would lie when
        // Canton actually committed (the case we see on devnet today).
        return
    }
    if transferErr != nil {
        _ = s.store.FailMempoolEntry(ctx, txHash, transferErr.Error())
        return
    }
    _ = s.store.CompleteMempoolEntry(ctx, txHash)
}

This alone won't make Activity show the currently stuck transfers (those need #278's reconciler to ask Canton "did this idempotency key actually commit?") — but it prevents new transfers from being incorrectly marked failed when Canton actually committed, and prevents the status from being lost altogether.

Full resolution still tracked elsewhere

• This issue: raise the timeout envelope (✅ fix: increased server timeout #280) + the detached-ctx defensive fix above
• Make eth_sendRawTransaction async (return tx hash immediately) #278: async submission + the actual reconciler that resolves stuck pending entries by querying Canton

I'll open a small follow-up PR for the detached-ctx change against main.