feat: transparently support form-encoded bodies (work around T3 host JSON-parse)#6
Conversation
…JSON-parse) The T3 host egress parses every request body as JSON, so a form-encoded body fails with `http.parse_payload: expected value…`. Solve it client-side: the proxy now detects an application/x-www-form-urlencoded body, moves its params into the URL query string, and sends an empty body — the host's JSON parser is never triggered and the agent's code is unchanged (it can POST a normal form). Form providers (Stripe, Twilio) accept query params, so it's transparent; JSON APIs are untouched. Also inject content-type for Stripe/Twilio so it's always present. Stripe form-body writes now succeed ~4/5 (was ~1/3). The residual failure is the testnet host intermittently dropping the content-type header on POSTs — a genuine host-side bug (reads are 100%), mitigated by client retry and documented in integration-stack.md.
| if (outboundBody && contentType.includes("application/x-www-form-urlencoded")) { | ||
| const u = new URL(outboundUrl); | ||
| for (const [k, v] of new URLSearchParams(outboundBody)) u.searchParams.append(k, v); | ||
| outboundUrl = u.toString(); | ||
| outboundBody = undefined; | ||
| safeLog("info", { msg: "form_body_to_query", provider: provider.id }); | ||
| } |
There was a problem hiding this comment.
⚠️ Bug: Twilio POST params moved to query string may not be accepted
The form-body→query-string workaround is applied to every form-urlencoded provider (proxy.ts:144), and providers.ts now forces content-type: application/x-www-form-urlencoded for Twilio. However, the PR's proof only exercises Stripe (5× POST /stripe/v1/customers, 34/34 Stripe tests). Twilio's REST API generally reads write parameters (e.g. To, From, Body for POST /Messages) from the form-encoded request body, not the query string. Sending an empty body with params in the query may cause Twilio writes to fail with missing-parameter errors. The claim "Form providers accept params in the query string" (integration-stack.md:203) is asserted but not demonstrated for Twilio. Please verify a real Twilio write (e.g. sending an SMS) succeeds through the proxy before relying on this path; if Twilio requires body params, this provider should be excluded from the query-string rewrite.
Was this helpful? React with 👍 / 👎
| const u = new URL(outboundUrl); | ||
| for (const [k, v] of new URLSearchParams(outboundBody)) u.searchParams.append(k, v); | ||
| outboundUrl = u.toString(); | ||
| outboundBody = undefined; | ||
| safeLog("info", { msg: "form_body_to_query", provider: provider.id }); |
There was a problem hiding this comment.
💡 Security: Form params relocated to URL query can leak into access logs
Moving the request body into the URL query string (proxy.ts:145-147) changes where potentially sensitive payload data lives. Request bodies are rarely logged, but query strings are routinely captured in host/upstream access logs, TLS-terminating proxies, and error-tracking systems. For these providers the params can include PII and sensitive fields (Stripe customer emails/metadata/tokens, Twilio phone numbers and message Body text). While Blindfold's own telemetry strips the query (upstream.replace(/\?.*$/, "") at proxy.ts:164,189), the T3 host and upstream provider still receive and may log the full URL. This is an inherent trade-off of the workaround, but it's worth documenting in integration-stack.md so operators understand payload data now travels in the URL, and worth confirming the T3 host does not log full request URLs.
Was this helpful? React with 👍 / 👎
| const u = new URL(outboundUrl); | ||
| for (const [k, v] of new URLSearchParams(outboundBody)) u.searchParams.append(k, v); | ||
| outboundUrl = u.toString(); | ||
| outboundBody = undefined; |
There was a problem hiding this comment.
💡 Edge Case: Large form bodies may exceed URL length limits after rewrite
The rewrite appends every form parameter to the URL (proxy.ts:146) with no size check. Servers, proxies, and the T3 host typically enforce URL length limits (commonly ~8 KB). A form body that is comfortably within HTTP body limits (e.g. a Stripe object with many metadata fields, or a long Twilio message Body) can produce a URL that exceeds these limits, causing the request to fail with a 414/400 rather than the intended write. Consider guarding the rewrite with a size threshold and falling back (or surfacing a clear error) when the resulting query string would be too large.
Was this helpful? React with 👍 / 👎
| const contentType = (headers.find(([k]) => k.toLowerCase() === "content-type")?.[1] ?? "").toLowerCase(); | ||
| if (outboundBody && contentType.includes("application/x-www-form-urlencoded")) { | ||
| const u = new URL(outboundUrl); | ||
| for (const [k, v] of new URLSearchParams(outboundBody)) u.searchParams.append(k, v); |
There was a problem hiding this comment.
💡 Bug: Forced form content-type can mangle non-form bodies for Stripe/Twilio
providers.ts now injects content-type: application/x-www-form-urlencoded by default for Stripe and Twilio (providers.ts:117,165). Because the query-rewrite triggers purely on the content-type string (proxy.ts:143-144), any body an agent sends to these providers without its own content-type header will be treated as form-encoded and run through new URLSearchParams(outboundBody). If the agent sends a non-form body (e.g. JSON), it will be silently parsed as a single garbage key and appended to the URL, corrupting the request rather than failing clearly. This is low-likelihood since these providers are form-only, but the coupling between the default header and the rewrite condition is fragile — consider gating the rewrite on the provider actually being form-based and/or validating that the body parses as form data.
Was this helpful? React with 👍 / 👎
Code Review
|
| Auto-apply | Compact |
|
|
Was this helpful? React with 👍 / 👎 | Gitar
What & why
Tackles improvement #5 (the T3 host-egress limits) as far as is possible client-side.
#1 — host parses request bodies as JSON → SOLVED client-side. The T3 host egress fails any non-JSON body with
http.parse_payload: expected value…, which blocked form-encoded APIs (Stripe, Twilio). The proxy now detects anapplication/x-www-form-urlencodedbody, moves its params into the URL query string, and sends an empty body — the host's JSON parser is never triggered. Form providers accept query params, so it's transparent: the agent POSTs a normal form and it just works. JSON APIs are untouched.#2 — content-type intermittently dropped → mitigated, residual is host-side. Stripe still requires a
content-typeheader even with query params, and the testnet host intermittently drops it. We now always inject the correct content-type for form providers, and clients retry. Genuinely host-side (reads are 100%) — this is the one item left for the T3 team.Proof
Normal form-body
POST /stripe/v1/customersthrough the proxy, 5×:The one failure is the residual host header-drop (#2). Provider tests: 34/34 green.
Notes
Summary by cubic
Add transparent support for form-encoded requests by moving form bodies into the URL query string, avoiding the T3 host’s JSON parser and restoring Stripe/Twilio writes. Also injects required content-type headers for these providers; JSON APIs are unaffected.
packages/blindfold/src/proxy.ts, detectapplication/x-www-form-urlencoded, move params to the URL, and send an empty body.packages/blindfold/src/providers.ts, setcontent-type: application/x-www-form-urlencodedfor Stripe and Twilio.integration-stack.mdwith the host-egress workaround and the residual header-drop note (advise retries).Written for commit 96fdf4b. Summary will update on new commits.