eth_sendRawTransaction forwarding fails with generic "error sending request" on follower nodes (v0.6.0, Ubuntu 24.04)

[mettin4]

Setup

• Arc node v0.6.0 (commit ab6b960)
• Ubuntu 24.04 on Hetzner dedicated server (Intel i7-8700, 128GB RAM, Helsinki)
• Running as a follower node (no validator key)
• Public RPC exposed via nginx reverse proxy at https://rpc.mtcrpc.org
• Read calls work fine, only eth_sendRawTransaction fails

Issue

Every transaction submitted via cast send or MetaMask fails with: Error: server returned an error response: error code -32603: error sending request for url (https://rpc.testnet.arc.network/)

Debug logs show: DEBUG rpc::eth: forwarding raw transaction to forwarder, DEBUG alloy_rpc_client::call: sending request method=eth_sendRawTransaction, DEBUG rpc::eth: failed to forward raw transaction err=error sending request for url

The error is generic with no underlying cause logged.

What I've verified works from the same server

• curl -X POST to all known forwarders returns valid responses (rpc.testnet.arc.network, rpc.quicknode.testnet.arc.network, rpc.drpc.testnet.arc.network)
• TLS handshake to forwarder succeeds via openssl s_client (valid Let's Encrypt cert, full chain verified)
• DNS resolves correctly via systemd-resolved, Cloudflare 1.1.1.1, Google 8.8.8.8
• Confirmed forwarder URL is reachable end-to-end with curl --http1.1

What I've tried (none of these helped)

• update-ca-certificates (already up to date, 219KB cert bundle)
• Setting SSL_CERT_FILE and SSL_CERT_DIR environment variables in the systemd service
• Disabled IPv6 system-wide (sysctl net.ipv6.conf.all.disable_ipv6=1)
• Replaced systemd-resolved stub resolver with direct nameservers (1.1.1.1, 8.8.8.8) in /etc/resolv.conf
• Removed --enable-arc-rpc flag
• Tried QuickNode, dRPC, and the default rpc.testnet.arc.network as forwarders, one at a time (multiple --rpc.forwarder flags crash the binary with exit code 2)
• Restarted execution layer multiple times

Reproduction

1. Build arc-node v0.6.0 from source on Ubuntu 24.04
2. Start execution + consensus layers as follower (no validator key)
3. Wait for sync
4. Send any transaction via cast send or MetaMask through the node's RPC

Expected behavior

Transaction is forwarded to the configured forwarder and submitted to the network.

Actual behavior

Forwarder request fails immediately with no underlying error detail beyond "error sending request for url".

Happy to provide additional logs or run any debugging commands. Verified the transaction was not silently submitted by checking balance after each failed attempt.

[Cassxbt]

I dug into this against v0.6.0 / ab6b9608e3f61128315b4ce6e10590d911057b9a and the forwarding path looks narrower than the current error suggests.

Verified from the dependency/source path:

• eth_sendRawTransaction forwarding is handled in Reth v1.11.3: it calls client.request("eth_sendRawTransaction", (rlp_hex,)).await and logs only %err on failure, so lower-level causes are currently hidden behind error sending request for url (...). Source: https://github.com/paradigmxyz/reth/blob/v1.11.3/crates/rpc/rpc/src/eth/helpers/transaction.rs#L96-L107
• The forwarder client is built through RpcClient::new_http(...), which uses Alloy HTTP transport backed by reqwest::Client::new(). Source: https://github.com/paradigmxyz/reth/blob/v1.11.3/crates/rpc/rpc-eth-types/src/tx_forward.rs#L18-L20
• The normal v0.6.0 arc-node-execution build has reqwest v0.12.28 with TLS enabled (rustls-tls, rustls-tls-native-roots, and rustls-tls-webpki-roots), so the standard source build path is not missing HTTPS support.
• A pinned probe using alloy-rpc-client = 1.7.3, alloy-transport-http = 1.7.3, and reqwest = 0.12.28 can reach all three public Arc forwarders from Ubuntu and gets the expected JSON-RPC decode error for an intentionally invalid raw tx, not a transport error.

The strongest reproduction I found for the exact top-level error is proxy environment leakage into the process. With the same pinned Alloy/reqwest stack, setting a bad HTTPS_PROXY reproduces:

error sending request for url (https://rpc.testnet.arc.network/)

The hidden source chain is then:

client error (Connect)
tunnel error: failed to create underlying connection
tcp connect error
Connection refused

This matters because curl from an interactive shell can pass while the systemd service running arc-node-execution still has a different environment. reqwest::Client::new() uses the default reqwest client behavior, and reqwest enables the automatic system proxy matcher unless explicitly disabled.

Could you check the live process and service environment for proxy/cert overrides?

PID="$(pidof arc-node-execution)"
sudo tr '\0' '\n' < "/proc/$PID/environ" | grep -Ei '^(HTTP_PROXY|HTTPS_PROXY|ALL_PROXY|NO_PROXY|http_proxy|https_proxy|all_proxy|no_proxy|SSL_CERT_FILE|SSL_CERT_DIR)='

sudo systemctl show <your-arc-execution-service> -p Environment
sudo systemctl cat <your-arc-execution-service>
systemctl show-environment | grep -Ei 'proxy|ssl_cert'

If any proxy variables are present in the process/service environment, a clean confirmation test would be to temporarily unset them for the service and retry eth_sendRawTransaction. On Ubuntu/systemd that should be done at the service level, not only in the shell where curl was tested.

Separate note on multiple --rpc.forwarder flags crash the binary with exit code 2: I reproduced that on v0.6.0, and it is a Clap argument-parse rejection before startup, not a runtime crash:

error: the argument '--rpc.forwarder <FORWARDER>' cannot be used multiple times

So I would treat that as expected CLI validation unless the intended behavior is to allow multiple fallback forwarders.