feat: Neon-compatible HTTP /sql endpoint

[MorganaFuture]

What kind of change does this PR introduce?

New HTTP endpoint.

What is the current behavior?

Supavisor speaks Postgres only over TCP. Serverless JS runtimes (Vercel, Cloudflare Workers) can't keep TCP pools between requests - every call pays a fresh SCRAM handshake. The WebSocket proxy at /v2 helps some clients but not the HTTP variant the JS ecosystem (@neondatabase/serverless, @vercel/postgres, drizzle-orm/neon-http) actually uses.

What is the new behavior?

POST /sql with the Neon wire format. Drop-in for @neondatabase/serverless:

  import { neon, neonConfig } from '@neondatabase/serverless'
  neonConfig.fetchEndpoint = (host) => `https://${host}/sql`
  const sql = neon('postgres://postgres.<external_id>:<pwd>@<host>/postgres')
  await sql`SELECT id FROM users WHERE email = ${email}`

The controller is itself a Postgres client of Supavisor - it dials 127.0.0.1:proxy_port_transaction via Postgrex, so every HTTP request reuses the existing ClientHandler -> pool -> DbHandler machinery. No changes to the TCP path.

Highlights:

• Single query and batched transaction (queries: [...]).
• Synchronous SCRAM-SHA-256 password check before pool checkout - bad-password attempts don't churn pools and feed the existing CircuitBreaker correctly.
• Per-(tenant, user, password_hash) Postgrex pool registry with idle-TTL eviction, invalidated on tenant updates via Supavisor.del_all_cache/1.
• PromEx metrics and per-error audit log.

Both gates are off by default: HTTP_SQL_ENABLED=false and per-tenant feature_flags.http_sql.

Additional context

Integration suite for the new endpoint is 101/101. A Node script at test/integration/js/neon_http_sql/index.js runs the real upstream @neondatabase/serverless driver against a live instance - release-gate smoke before each ship.

TODO: cache_statement for repeat SQL, per-IP rate limit, JWT-auth path (rejected with 401 today).

[MorganaFuture]

Hey @v0idpwn, @mentels I have opened PR for #152, Neon-compatible HTTP /sql endpoint. Would love a look when you've got a time. It is a chunky PR, though I tried to keep the commits small and in a sensible reading order.

[v0idpwn]

Hi, @MorganaFuture, thanks for the PR, that's a lot of work!

It will take some time to review. I'll do a first pass today, but there will probably be more iterations.

[v0idpwn]

Again, thanks for putting the effort to write this!

I think doing this via Postgrex is a mistake, and makes the code more complex than it should be. Have you considered doing it via an alternative client worker? Imagine the following:

(1) HTTP handler receives request
(2) Spawn a one-off client handler (I think it would not be possible to reuse the ClientHandler module here, but you may implement an alternative ClientHandler inspired on it, e.g.: HTTPClientHandler) that does the regular operations: spawn pool if it doesn't exist (if it does, reuse it), check out a connection, and send the SQL to the database. This will also help us to enforce limits (e.g.: client handler limit)
(3) You will be able to use the simple query protocol here, so you won't need to coerce parameters, etc.
(4) I'm not 100% sure yet of how the query results should be handled, either:

• they are forwarded directly to the client
• they are sent to the HTTP handler which performs necessary transformations and sends them to the client

Another alternative to explore: instead of a custom client handler, reuse the existing one like the WS module does.

[v0idpwn]

Did you have issues running them via ExUnit, like the existing Neon websocket adapter?

[MorganaFuture]

Nope, kept it manual during the dev process. Will fold it into ExUnit as part of the rewrite

[MorganaFuture]

Again, thanks for putting the effort to write this!

I think doing this via Postgrex is a mistake, and makes the code more complex than it should be. Have you considered doing it via an alternative client worker? Imagine the following:

(1) HTTP handler receives request (2) Spawn a one-off client handler (I think it would not be possible to reuse the ClientHandler module here, but you may implement an alternative ClientHandler inspired on it, e.g.: HTTPClientHandler) that does the regular operations: spawn pool if it doesn't exist (if it does, reuse it), check out a connection, and send the SQL to the database. This will also help us to enforce limits (e.g.: client handler limit) (3) You will be able to use the simple query protocol here, so you won't need to coerce parameters, etc. (4) I'm not 100% sure yet of how the query results should be handled, either:

* they are forwarded directly to the client

* they are sent to the HTTP handler which performs necessary transformations and sends them to the client

Another alternative to explore: instead of a custom client handler, reuse the existing one like the WS module does.

@v0idpwn Thanks for feedback! Before I start to the rewrite, I have two question:

The first is DbHandler.checkout. The way ClientHandler calls it, the caller hands over a TCP socket and DbHandler writes backend bytes straight back into it. HTTP doesn't have a socket on the receive side the response bytes have to land in a process that parses the Postgres wire response and turns it into a Neon JSON body. I can see two paths. Either wrap that process in something socket-shaped so DbHandler can drive it the same way it drives a real socket, or extend checkout so the caller can opt into receiving frames as plain Erlang messages and leave the existing TCP path alone.What do you think?

The second is the simple-query idea. The Neon driver sends query and params as separate JSON fields, and ?Q doesn't carry params, so going pure simple-query means inlining params into the SQL with escaping, which puts the injection boundary inside Supavisor. If we use extended protocol with text-format params instead (no binary, no typing), Postgres parses each string server-side and ParamCoercer still goes away. That feels like the right tradeoff.

A couple of things I sorted out already so we don't go back and forth. The value encoder isn't going away completely bytea and arrays come out of libpq's text format in a different shape than Neon's JSON wants, so some reshaping has to stay. The limit gap is real; my pools currently sidestep Manager.subscribe, and on the rewrite I'd register through it so HTTP clients land in the same accounting as everything else. And ws_proxy is just a byte relay to the proxy port. I'm reading "reuse like WS does" as reuse of the internal checkout, not the WS pattern itself. Say if that's wrong.

[v0idpwn]

The first is DbHandler.checkout. The way ClientHandler calls it, the caller hands over a TCP socket and DbHandler writes backend bytes straight back into it. HTTP doesn't have a socket on the receive side the response bytes have to land in a process that parses the Postgres wire response and turns it into a Neon JSON body

Currently, DbHandler handles either {:ssl, sock} or {:tcp, sock}. Could we pass {:proc, pid}` and have it send as messages? 🤔

The Neon driver sends query and params as separate JSON fields

My bad, you are right, we need to use the extended query protocol here! But, if we format the messages ourselves instead of relying on Postgrex, the extneded query protocol supports text-based parameters, see this

[MorganaFuture]

Currently, DbHandler handles either {:ssl, sock} or {:tcp, sock}. Could we pass {:proc, pid}` and have it send as messages? 🤔

yep, works cleanly. sock_send is the single fan-out and client_sock is the only field holding the variant, so it's one extra clause + letting that field carry {:proc, pid}. tcp/ssl paths don't move.

My bad, you are right, we need to use the extended query protocol here! But, if we format the messages ourselves instead of relying on Postgrex, the extneded query protocol supports text-based parameters, see this

Agreed, text-format Bind is the play. Starting on rewrite. Can you please assign issue #152 to me?

[MorganaFuture]

@v0idpwn Could you please take a look?